Final Post

My project has finally come to a close, and to recap here's a quick video of what I implemented (it pretty much combines videos from some of the previous posts):



I also went back and updated posts in the blog to better highlight my progress throughout the semester. I found myself often saving drafts while waiting to upload a video, only to forget to publish the post, so I cleaned all that up.

Anyway, it's been fun!

Cleaning Things Up

After the presentation, I spent some time cleaning up and my code to make it easier to read/edit. I kind of restructured it so parts of the code can hopefully be useful for functionality even outside of overlays.

I also went over my implementation of methods from previous works to test if it worked ok. I actually found some mistakes in my implementation and fixed those.

I'm going to to find some more upper body motions and generate more overlays before the project is finally due.

Generating and Sorting the Overlays

I've been generating overlays using my script throughout the project. Now that everything relevant is implemented (spatial alignment, time alignment, physics checks) I can generate the final set of overlays. The script will automatically sort the overlays based on whether or not they passed the physics checks.

I have to look at each individual overlay and determine whether or not it looks natural. This is very subjective and there are a lot of borderline cases where I'm not sure what to do...

The goal is to sort all the overlays into sets of Natural & Physically Correct, Unnatural & Physically Correct, Natural & Not Physically Correct, Unnatural & Not Physically Correct. Doing so will give a broad overview of how well the physics checks are working in terms of classifying an overlay as natural or not.

Based on the results of the generated overlays, I think the physics checks do a solid job of filtering out the bad, unnatural overlays as failing the checks. However, there are some overlays that fail the checks but look natural. There are actually a lot of overlays that pass the physics checks but look unnatural. So from an even rudimentary analysis we can see that physical correctness is not enough to guarantee that an overlay looks natural.

Changing Speeds Based on Center of Mass

When demoing my project to Dr. Badler, he offered the suggestion of correcting the motions when the center of mass of the character changes significantly after creating an overlay. For example, if the character is running fast, he is naturally leaning forward. If you want to transplant an upper body motion on to this and use spatial alignment, the character will be leaning forward too much for it to look natural or even possible. We can use our physics checks to determine the character's center of mass throughout a motion. By comparing how the center of mass position changes in the upper body motion before and after transplanting it to the lower body motion, I can identify whether the character will be leaning forward too much or leaning back too much. If the change is enough that the character will need to slow down to maintain a close enough center of mass to before the transplant, I interpolate the lower body motion with a slower speed motion. Here is an example of what an overlay looks like before and after the speed is corrected based on the change in center of mass:



This is an admittedly rough implementation more useful as an example. I think this would be especially useful in games if the player wants to perform an action, but the character needs to change speeds automatically to perform that action properly.

Updated Note: For my presentation, I switched the way I did spatial alignment to align the lower body motion to the upper body motion. However, the approach I took initially is better I think (aligning upper body motion to lower body motion). As a result, the example I used in the presentation is the opposite of what is here, but I think the approach here is the correct one.

Game Environment

I created a game environment in OGRE. This preliminary version does not contain much but, serves as another tool to visualize motions. I used one of our skinned character models and exported it to OGRE from Maya. I linked the game code with the code for our framework to allow the character to perform any motion easily. Here is what it looks like:



Obviously there is a lot of room for improvement. I may work on adding more to the game environment or adding some level of user control for the character. When I first planned the project, I thought I would use a motion graph and allow the user to fully control the character. However, I figured that I would not have enough time to include everything so I thought just loading a motion would be sufficient.

Time Alignment

Another element in creating overlays is making sure both motions are in phase at the same time. For example, if you have two walking motions, you want both motions to be in the same phase of the walk cycle at each frame before doing a transplant.

Coincidentally, a lot of overlays already look as if they are in phase when doing a regular transplant. If we transplant the upper body of a character standing to the lower body of a character slowly walking, it does not look that bad. Problems however can occur when dealing with motions where the character is moving at different paces.

There were two methods I thought about to implement time alignment. One is dynamic time warping, which is used in the "Splicing Upper-Body Actions with Locomotion" paper by Heck, Kovar, and Gleicher. The other is uniform time scaling. I decided to go with dynamic time warping because I could also use it to check for similarity between motions (with a metric proposed in "Enriching a Motion Collection by Transplanting Limbs" by Ikemoto and Forsyth). The similarity checks was something I thought about to consider when auto generating overlays (I'll probably post more on that later).

The dynamic time warping approach computes point clouds for the lower bodies of both motions and creates a distance matrix, using an approach from the "Motion Graphs" paper by Kovar and Gleicher. I found that our framework had a similar metric for computing distances between frames so I decided to use that.

First, I had to make sure that the first frames of both motions were in phase. I compared the first frame of the shorter motion with frames of the longer motion to determine the most similar frame, and cropped the longer motion so both initial frames were similar enough.

I then looked at the dynamic programming method in "Flexible Automatic Motion Blending with Registration Curves" also by Kovar and Gleicher (at this point it seems like they do just about every relevant paper!) However, the paper didn't actually describe the algorithm so I had to look through a lot of other papers (many of which contained different elements that I didn't need) to find the correct method for our purposes. When computing the matrices, I also had to ensure that too many frames didn't map to a single frame (which guarantees a good slope in the time alignment curve). I set the limit so that there are at most 3 frames mapping to one. After computing the cost matrices, I computed the optimal path starting from the first frame and assigned each new frame based on the frame to frame mapping.

Here is an example of how this time alignment can improve an overlay:



Of course, time alignment is only useful when you have two motions that are moving similarly.

I may look into implementing uniform time alignment as well, if time permits. (To reiterate, this method was dynamic time warping.)

Spatial Alignment Results

After a lot of debugging and testing, I believe I have spatial alignment implemented correctly. I would say the difference between using spatial alignment and not using it is noticeable. In general though, the spatial alignment method can preserve details of the lower body motion before the transplant. But is that necessarily a good thing? Here are some of the results:



We can choose to try to align the upper body motion with the lower body motion (as done in the video above) or vice versa. From my observations, choosing either or does not improve all overlays and which is better depends on the overall postures in both motions and what the particular goal is. I'm assuming in most cases we want the upper body motion to adapt to whatever the lower body is doing so I'm leaving it as is for now. I'll have to look into possible ways to fix what is happening above.

I wanted to implement spatial alignment to solve the twitching problem caused because of the lack of correlation between the upper and lower body. Although it doesn't solve that problem in all cases, here's an example where spatial alignment can alleviate the twitching problem:



Notice that the alignment eliminates the unnecessary lean to the left in the original overlay. However, noise (in the form of very rapid twitching) is prevalent in a lot of the overlays. Perhaps we'll have to look into methods in filtering out the noise to produce better results.

Spatial alignment does not correlate the lower body with the new upper body in all cases. We have a lot of upper body motions that involve a character doing an action while standing. Even with alignment, these upper body motions look awkward when transplanted to a character that's running.

I'm currently working on time aligning the motions before transplanting limbs, although that presents some issues as we may be combining motions where the character is standing, running, or doing some other type of motion. I'm also looking into research for quantifying the naturalness of a motion, which could be useful when generating overlays. Lastly, I'm working on implementing the game environment in OGRE to visualize the overlays on a skinned character.