ENLIGHTEN3D

By Christian Laforte

It’s been a over month since Google launched the browser plug-in for Google Earth, combining the best of Google Maps (fast, easy, in the browser, extendable through javascript) with the capability to navigate in a 3D terrain.

So does the 3D capability really add a lot over the regular Google Maps? It’s still too early to tell, but a few examples show the potential applications in education, entertainment and planning.

The most visually compelling example I’ve seen so far comes from Bjorn Sandvik’s ThematicMapping blog:

In a glance, you can see in which countries infant mortality is a critical problem. A color legend alone doesn’t fire the imagination the same way.

Another example, the Google Monster Milktruck, is kind of fun:

This mini-game allows you to drive the milk truck around. Unfortunately, many limitations of the Google Earth engine become apparent, especially the lack of collision detections with walls.

A third example, from GolfNation’s blog, allows you to see a golf course in 3D:

This example demonstrates the main problem with Google Earth right now: for the 3D capability to be worthwhile, we need more 3D content. Trees, cars and buildings look like they are painted on the ground, because we don’t have a 3D representation. We’re basically just looking at 2D data (satellite imagery) from a different perspective.

Google and Microsoft are apparently working hard on this problem of reconstructing buildings and landmarks. Feeling Software is also investing in 3D reconstruction from images… but we’re taking a different strategy, that hopefully will put us one step ahead of these giants, in one promising niche. Incidentally, our Feeling 3D Engine also supports KML and KMZ, along with geo-referencing and geographic measurements.

Incidentally, there are other 3D GIS (geographic information systems) that work on the web. One interesting example comes from Korea, according to this informative ZDNet article:

If you know of other compelling examples of 3D use in GIS, by all means, reply to this post!

By Christian Laforte

Chances are you have already heard and are waiting to play Spore, arguably the most ambitious video game ever. Spore allows the player to easily model and control the evolution of an alien species, including an original 3D, fully-animated creature. The possibilities are infinite: your creature can have multiple heads, zero or dozens of legs, multiple sets of arms or claws, a mouth in the middle of its belly, etc. Here’s an example that you can reproduce using the free Creature Creator tool.

For the game to be enjoyable, each unique creature must be animated to show feelings and intent. So how do you make a two-headed worm look excited? And how do you animate a thirteen-legged, three-armed, two-mouthed monster running happily? This would already be challenging for an experienced 3D animator. But because the user can change his creature anytime, the animations must actually be adapted on the fly.

So how did they do it? The EA engineers have opened up a lot of their secrets, most of which are collected on Chris Hecker’s page.

Their most detailed research paper will soon be published at Siggraph 2008:

Real-Time Motion Retargeting to Highly Varied User-Created Morphologies

Chris Hecker, Bernd Raabe, Ryan W. Enslow, John DeWeese (Maxis/Electronic Arts), Jordan Maynard (Trion World Network), Kees van Prooijen (Total Immersion Software)

If you’re a game or 3D graphics developer, by all mean, read this paper. As far as Siggraph papers are concerned, it’s refreshingly practical and accessible. About half of the paper concerns special cases that are only applicable if you try to animate alien characters, e.g. how to handle with look-at constraints for two-headed characters.

Still, there are a lot of techniques applicable to any game or animation system, including a much improved IK (inverse kinematics) solver which could help produce more believable animations in games. In case you don’t know, an IK solver is used to pose the character according to some constraints or goals, e.g. bring one arm to the food, bring the food in the mouth, shake hand with the hand of another character. The IK system in Spore is designed for real-time and elegantly handles conflicting goals. The same system could be applied to complex dynamic animations on human characters, e.g. animate the character aiming a gun, running and reaching for an object at the same time, in the most natural way.

Their animation retargeting technique should also help artists be more productive, by creating animations once and having the computer automatically adapt it to characters with longer legs or arms.

So how well does it work, and will it be fun? I tried the Creature Creator tool. I found it exciting to create my first two creatures. The authoring tool is extremely simple, and there’s a lot of smart usability design in there, like automatic mirroring of body parts, a simple but useful idea that isn’t as easily supported in Maya and 3ds Max.

The animations are fun to watch and listen to, especially the first ten minutes. The gait animation (walking and running cycles) is really cool to watch in action. Even the weirdest character ends up conveying fear, happiness and excitement effectively.

After ten minutes I started noticing small problems here and there. The keyboard movement is annoyingly jerky. The animation and simulation system doesn’t check for interpenetration… so one leg of my creature frequently passed through the torso, breaking the illusion. Sometimes the heads even disconnected from the rest of the body. Finally, the system doesn’t model the mass of the body parts, so you can end up with creatures that look like they should topple, but mysteriously hang in balance.

I suspect that some of these problems will be addressed in the game, especially the keyboard control jerkiness.

So how much effort would it take to reproduce this complex animation system? It all depends on what you want to achieve and where you start with. Hecker and his Spore colleagues made their own animation authoring tool, which makes sense since they wanted to integrate the functionality right inside their game. If all you’re interested in is some pre-processed animation retargeting and improved IK, you could save time by building the functionality as a plug-in to Maya or your favorite animation package… although you’ll still need to implement the runtime IK evaluation if you want to adapt the animation in the game.

If you’re interested in supporting unusual creatures, with multiple heads, arms in unusual locations, etc. then you could implement a variant of Spore’s selection menu and branching predicates. I suspect this could save a lot of time in pre-visualization for CG movies, when animating creatures like these:

Candlestick character from Disney’s Beauty and the Beast

Cerberus mythological character, unknown artist

If we were asked by a client to implement some of this functionality, we would likely start from the Feeling Engine or a Maya plug-in. The Maya plug-in would be a natural approach if the final desired result is an offline animation, to be touched up by an experienced animator. Otherwise, the Feeling Engine already supports complex character animations, physics and 3D manipulators, and easy back-and-forth with Maya or 3ds Max, so this solution would be most appropriate for a stand-alone tool.

By Christian Laforte

Do you know what CUDA and OpenCL stand for and how they could make your computer 50 times faster? If so, you can safely jump to the “Ending the mess” section below. Otherwise read on for a gentle introduction.

A computer has two important processing units: the CPU and GPU. Think of them as the two brothers in Rain Man.

The GPU is the ultimate autistic savant. He’s really, really good at counting stuff and doing a lot of complex math at the same time. For example, he can multiply two long sequences of numbers in his head, faster than you can type it in a calculator. But ask him to do something he’s not used to like buying milk and he’ll just ignore you or throw a fit. He only listens to people who know him well. Even if you spend years learning to communicate with a given autistic savant, chances are you’ll have to start anew when you meet a new one.

The CPU is your regular guy. He can do all kinds of stuff that the savant can’t. He goes along well with everybody, as long as they speak English. If he learns to take advantage of the savant, the two of them can do amazing things like count cards at Poker.

In other words, the GPU is natural at some operations that involve repetitive calculations, like those necessary for drawing 3D graphics and doing basic image manipulation. It can do those operations hundreds of times faster than regular CPUs. The high performance comes at the cost of ease of programming. As long as you stick with basic 3D graphics, it’s pretty easy. But say you want to make your financial application run 50 times faster, or making your protein folding simulator run a hundred times faster. It can be done, but for every ten thousand programmers out there, you’ll be lucky if you find one GPU expert capable of achieving that. (Shameless hint: you’ll have a much higher chance to find what you’re looking for at Feeling Software.)

The main GPU vendors — NVIDIA, AMD and Intel — have created new programming languages in the hope to simplify this process for non-GPU experts. For the time being NVIDIA seems to be leading the pack with CUDA. AMD has the Stream SDK. Intel provides Ct. Apple has OpenCL. And Microsoft has… well Microsoft doesn’t have anything yet, it’s too busy introducing new bugs in Vista.
History lesson

Wouldn’t it be nice if everyone could play nice together, if we could get any computing-intensive application to run ten or a hundred times faster without having to deal with all these vendor-specific languages? A potential solution is looming. Before I give you the answer, a short history lesson:

Back when the dinosaurs roamed the earth, in the early 1990s, there were dozens of 3D graphics workstation vendors whose name have long been forgotten who tried to dominate the 3D graphics industry. One of them, SGI, offered generously to transform its superior technology into an open standard called OpenGL. With OpenGL, programming simple 3D graphics algorithms on different hardware became easy.

Shortly after, SGI tragically lost its mind. Every smart graphics engineers fled the company to join upstarts like NVIDIA and ATI (bought by AMD recently). On its death bed, SGI gave up OpenGL and allowed a standard group called Khronos to take it over, where is has since evolved at a moderate but consistent rate. Khronos also maintains other 3D standards like COLLADA.

Ending the mess

Today Khronos announced that it wants to repeat the exploit. This time, they are starting a Compute Working Group so NVIDIA, AMD, Intel and many more can try to agree on a cross-platform standard, i.e. a programming language that will run super fast on multi-core CPUs and on GPUs alike. History shows that this is a good move, that can make it possible for new applications to come to life, applications that would have been too complicated to implement otherwise.

I have seen a couple of these standardization cycles in action, and since I’m not participating in this working group (so far) I will simply speculate from past experience and from the little that has been publicly stated so far. (Feel free in the comments section!) I think there are important questions that will make or break this initiative:

- Are the main players in a collaborative mindset, or are trying to dominate each other? In previous cycles (e.g. GLSL) Apple acted as the mediator, since they badly needed that technology and it had a strong business influence on NVIDIA and ATI. Intel didn’t have a strong participation. This time Intel is very active in 3D graphics. ATI/AMD is in a tough spot where they badly need to get back some market share, maybe through deals with Apple. I’m hopeful that there will be enough collaboration to create a decent standard, with unusual high-performance features continuing to be exposed only in the proprietary standards.

- Typically, the resulting standard is strongly inspired by one precedent. OpenGL was derived from IrisGL from SGI. COLLADA came straight from Sony. Arguably GLSL was inspired by early versions of NVIDIA Cg and a long line of predecessors. This time around, Apple apparently likes NVIDIA’s CUDA which is likely to form the basis of their OpenCL. OpenCL would be the right name for the resulting standard, if the other players can accept it. So my bet is on CUDA rechristened as OpenCL, with bits of Intel Ct thrown in to keep everyone happy. AMD will keep its low-level interface and will implement the high-level standard on top.

- How long will it take for a something usable to come out of this? In the past, such standards have typically taken at least a year, often because modifications were necessary to make the hardware compatible. In this case, it sounds like the technology will primarily be software-based (e.g. compilers), so it could happen much faster if a majority of the vendors accept to play nice. The deciding factor here will be the chair of the working group. A diplomatic yet decisive chair can make things happen ten times faster than a less decisive or a visibly biased one. Since I don’t know yet who is chairing the group, I will refrain from commenting publicly.

[digg=http://digg.com/programming/Making_CPU_and_GPU_play_nice_together_ENLIGHTEN3D]

By Christian Laforte

Is beauty is in the eye of the beholder? Or is this just wishful thinking?

Psychologists have long known that infants are more attracted to attractive faces. They stare at them longer. Evidences are strong that humans are instinctively drawn to smooth skin and symmetric faces.

So what do you do if you’re not as pretty as you’d wish? Exercising, creams and plastic surgery only produce temporary results and take effort and money. Instead, I recommend a long-term solution: change the evidence, so you can brag about how you look to your virtual friends, or you can lament how beautiful you looked ten years ago. What if you could, in one click, make pictures of yourself, your family and your friends look like supermodels?

But isn’t this already possible? If you have money, time and talent at your disposal, you can hire good make-up and Photoshop artists to have a beautiful portrait:

Unfortunately, ugly people earn less than pretty people. So people like me who need this most can’t afford it.

Fortunately, computer graphics researchers are working on this problem. Tommer Leyvand and his colleagues have invented a new technique, Beautification, to enhance the aesthetic appeal of human faces, using a combination of machine learning and computer graphics algorithms.

Data-driven enhancement of facial attractiveness

Tommer Leyvand, Daniel Cohen-Or, Gideon Dror (Tel Aviv University), Dani Lischinski (The Hebrew University)

As noted by the authors, the applications are obvious and numerous. They include motion picture special effects, advertising, dating services and creating prettier avatars for virtual worlds. They also mention the interesting idea that every shapes (e.g. the front of a car) could be made more beautiful by making it look more like a pretty human face.

Their algorithm produces very cool results:

Before beautification:

… and after:

Still, don’t get excited too fast. Their approach is promising but there are still lots of limitations.

The algorithm uses pictures of faces, annotated with beauty scores. The faces are carefully taken, similarly to passport photographs, so the algorithms can operate in 2D. So it will take more research before they can apply their beautification to any of your old pictures, or before they can leverage the accumulated wisdom from the very scientific Hot Or Not web site.

Some of the “beautified” images also end up looking creepy. I might be wrong but I think that some of the eyes end up with different sizes of iris and pupils, something humans are very sensitive to. (Dilated pupils are a sign of sexual attraction.)

The approach is only demonstrated on Caucasian, adult people. The authors suggest that training the algorithm on populations of different races might help, but it’s still not clear how the algorithm could be extended to very diverse populations. I think more research would be necessary to deal with my daughter, a half-Japanese, half-Caucasian baby.

The algorithm focused on the geometric (i.e. shape) of the face. It doesn’t attempt to smooth out blemishes or eliminate acne. Maybe this could be fixed through other computer graphics techniques, e.g. bilateral filtering. Click on the following image to see what a difference bilateral filtering can make:

To come back to the beautification algorithm, instead of trying to make the face look more similar to a given pattern, I think a better approach would be more subtle: focus on correcting obvious asymetries, combined with the bilateral filter.

PS: There are other techniques out there to make faces pretty.

This approach reportedly averages faces, then exagerates the feminity: Unfortunately I couldn’t find any screenshot.

Do you know of other cool techniques to make faces prettier? Or can you think of unusual applications or markets that could benefit from this kind of technology? If so, let us know! We’d love to have an excuse to experiment with these algorithms… to make our faces the world a prettier place.

[digg=http://digg.com/programming/The_Beauty_Algorithm_Can_Computers_Make_You_Pretty]

By Joshua Koopferstock

We haven’t ever posted a job on a blog before, but in the spirit of trying new things, I figured I’d give it a go.

Feeling Software (that’s us) is looking for a computer vision software developer to join our team in dazzling Montreal, Canada. In this position, you will be working directly on our breakthrough 3D reconstruction technology that will allow the average Joe to reconstruct 3D interiors in five minutes. Think Microsoft PhotoSynth, but much faster, for smaller spaces, and creating photo-realistic 3D models.

If you are interested, or you know someone who is, come check out the full posting on our website, or send us an e-mail to hiring@feelingsoftware.com.

By Joshua Koopferstock

Reaching from 3D displays, web-based 3D, and 3D modeling to new advanced computer vision algorithms, the 3D world is vast and moving at an incredible pace. Innovation abounds in the 3D world, and if you don’t lower your head and sprint along with it, you’re probably going to see it pass you by.

Think of us as your new pair of running shoes. Our goal with the ENLIGHTEN3D blog is to explore the diverse reaches of the 3D industry and bring together sectors that have been drifting far apart. We’re doing this because we believe, for example, that the algorithms created by Noah Snavely (lead researcher on PhotoSynth) have an impact on the business of Linden Lab (founders of Second Life virtual world), and we want to talk about it. We see companies using some of the simplest 3D technologies to make a disproportionately big splash (think Apple’s iPod album scrolling), and other companies trying to use cutting edge 3D tech and not making the splash they’d hoped for (poor My Virtual Model).

We’re going to discuss how they did it, why it’s working or it’s failing, and where it might lead. And we’re doing it both from a business and technology slant. The ENLIGHTEN3D blog enlists the contributions of two authors, mashing together their respective technical know-how and marketing savvy.

This blog is for anyone who is constantly amazed by the world of 3D and its potential. We certainly are. If you are a computer vision specialist, this blog is for you. If you’re like me and programming code means nothing to you, then this blog is for you too. We are going to discuss 3D business and technology in a way that any geeky business person or literate programmer (just because you can read C++ doesn’t mean you can read. Zing!) can understand. We hope that our readers come with diverse backgrounds and will set us straight when we get too tech-speak or our business ideas start to enter the “only in Star Trek” realm.

That blur that just passed you was the whole field of 3D, so lace up those new shoes and get running! If you’re falling behind the pack, don’t worry; we’ll be here to poke you in the rump and get you up to speed.