Generating and Sorting the Overlays

I've been generating overlays using my script throughout the project. Now that everything relevant is implemented (spatial alignment, time alignment, physics checks) I can generate the final set of overlays. The script will automatically sort the overlays based on whether or not they passed the physics checks.

I have to look at each individual overlay and determine whether or not it looks natural. This is very subjective and there are a lot of borderline cases where I'm not sure what to do...

The goal is to sort all the overlays into sets of Natural & Physically Correct, Unnatural & Physically Correct, Natural & Not Physically Correct, Unnatural & Not Physically Correct. Doing so will give a broad overview of how well the physics checks are working in terms of classifying an overlay as natural or not.

Based on the results of the generated overlays, I think the physics checks do a solid job of filtering out the bad, unnatural overlays as failing the checks. However, there are some overlays that fail the checks but look natural. There are actually a lot of overlays that pass the physics checks but look unnatural. So from an even rudimentary analysis we can see that physical correctness is not enough to guarantee that an overlay looks natural.

Changing Speeds Based on Center of Mass

When demoing my project to Dr. Badler, he offered the suggestion of correcting the motions when the center of mass of the character changes significantly after creating an overlay. For example, if the character is running fast, he is naturally leaning forward. If you want to transplant an upper body motion on to this and use spatial alignment, the character will be leaning forward too much for it to look natural or even possible. We can use our physics checks to determine the character's center of mass throughout a motion. By comparing how the center of mass position changes in the upper body motion before and after transplanting it to the lower body motion, I can identify whether the character will be leaning forward too much or leaning back too much. If the change is enough that the character will need to slow down to maintain a close enough center of mass to before the transplant, I interpolate the lower body motion with a slower speed motion. Here is an example of what an overlay looks like before and after the speed is corrected based on the change in center of mass:



This is an admittedly rough implementation more useful as an example. I think this would be especially useful in games if the player wants to perform an action, but the character needs to change speeds automatically to perform that action properly.

Updated Note: For my presentation, I switched the way I did spatial alignment to align the lower body motion to the upper body motion. However, the approach I took initially is better I think (aligning upper body motion to lower body motion). As a result, the example I used in the presentation is the opposite of what is here, but I think the approach here is the correct one.

Game Environment

I created a game environment in OGRE. This preliminary version does not contain much but, serves as another tool to visualize motions. I used one of our skinned character models and exported it to OGRE from Maya. I linked the game code with the code for our framework to allow the character to perform any motion easily. Here is what it looks like:



Obviously there is a lot of room for improvement. I may work on adding more to the game environment or adding some level of user control for the character. When I first planned the project, I thought I would use a motion graph and allow the user to fully control the character. However, I figured that I would not have enough time to include everything so I thought just loading a motion would be sufficient.

Time Alignment

Another element in creating overlays is making sure both motions are in phase at the same time. For example, if you have two walking motions, you want both motions to be in the same phase of the walk cycle at each frame before doing a transplant.

Coincidentally, a lot of overlays already look as if they are in phase when doing a regular transplant. If we transplant the upper body of a character standing to the lower body of a character slowly walking, it does not look that bad. Problems however can occur when dealing with motions where the character is moving at different paces.

There were two methods I thought about to implement time alignment. One is dynamic time warping, which is used in the "Splicing Upper-Body Actions with Locomotion" paper by Heck, Kovar, and Gleicher. The other is uniform time scaling. I decided to go with dynamic time warping because I could also use it to check for similarity between motions (with a metric proposed in "Enriching a Motion Collection by Transplanting Limbs" by Ikemoto and Forsyth). The similarity checks was something I thought about to consider when auto generating overlays (I'll probably post more on that later).

The dynamic time warping approach computes point clouds for the lower bodies of both motions and creates a distance matrix, using an approach from the "Motion Graphs" paper by Kovar and Gleicher. I found that our framework had a similar metric for computing distances between frames so I decided to use that.

First, I had to make sure that the first frames of both motions were in phase. I compared the first frame of the shorter motion with frames of the longer motion to determine the most similar frame, and cropped the longer motion so both initial frames were similar enough.

I then looked at the dynamic programming method in "Flexible Automatic Motion Blending with Registration Curves" also by Kovar and Gleicher (at this point it seems like they do just about every relevant paper!) However, the paper didn't actually describe the algorithm so I had to look through a lot of other papers (many of which contained different elements that I didn't need) to find the correct method for our purposes. When computing the matrices, I also had to ensure that too many frames didn't map to a single frame (which guarantees a good slope in the time alignment curve). I set the limit so that there are at most 3 frames mapping to one. After computing the cost matrices, I computed the optimal path starting from the first frame and assigned each new frame based on the frame to frame mapping.

Here is an example of how this time alignment can improve an overlay:



Of course, time alignment is only useful when you have two motions that are moving similarly.

I may look into implementing uniform time alignment as well, if time permits. (To reiterate, this method was dynamic time warping.)

Spatial Alignment Results

After a lot of debugging and testing, I believe I have spatial alignment implemented correctly. I would say the difference between using spatial alignment and not using it is noticeable. In general though, the spatial alignment method can preserve details of the lower body motion before the transplant. But is that necessarily a good thing? Here are some of the results:



We can choose to try to align the upper body motion with the lower body motion (as done in the video above) or vice versa. From my observations, choosing either or does not improve all overlays and which is better depends on the overall postures in both motions and what the particular goal is. I'm assuming in most cases we want the upper body motion to adapt to whatever the lower body is doing so I'm leaving it as is for now. I'll have to look into possible ways to fix what is happening above.

I wanted to implement spatial alignment to solve the twitching problem caused because of the lack of correlation between the upper and lower body. Although it doesn't solve that problem in all cases, here's an example where spatial alignment can alleviate the twitching problem:



Notice that the alignment eliminates the unnecessary lean to the left in the original overlay. However, noise (in the form of very rapid twitching) is prevalent in a lot of the overlays. Perhaps we'll have to look into methods in filtering out the noise to produce better results.

Spatial alignment does not correlate the lower body with the new upper body in all cases. We have a lot of upper body motions that involve a character doing an action while standing. Even with alignment, these upper body motions look awkward when transplanted to a character that's running.

I'm currently working on time aligning the motions before transplanting limbs, although that presents some issues as we may be combining motions where the character is standing, running, or doing some other type of motion. I'm also looking into research for quantifying the naturalness of a motion, which could be useful when generating overlays. Lastly, I'm working on implementing the game environment in OGRE to visualize the overlays on a skinned character.

Implementing Spatial Alignment

First, let me recap why spatial alignment is important: when we transplant the upper body from one motion to the lower body of another, it is likely there will be noticeable problems in the form of twitching unless both motions were similar enough. This is because the upper body is usually correlated with lower body movements, and transplanting destroys this correlation.

Earlier in my project, I found a quick way to get rid of the twitching was to not transplant the joint above the root, the lowerback joint, which is responsible for rotation about the torso. Although excluding the lowerback joint eliminates the twitching, it sacrifices some rotation. A solution to this is proposed in "Splicing Upper-Body Actions with Locomotion" by Heck, Kovar, Gleicher [2006].

Following their method, called spatial alignment, we should compute a rotation for the lowerback joint that best aligns the shoulders and spine of the motion we use for the upper body with the motion for the lower body. The orientation is found by computing point clouds and solving for the closed form solution in "Closed-form solution of absolute orientation using unit quaternions" [Horn 1987].

Although it sounds simple enough, I've encountered a number of issues while trying to implement it. The closed form solution needs to compute eigenvalues of a system, so I had to find the appropriate matrix library to do that. I then implemented the solution and had to run many unit tests to ensure that it was implemented correctly.

The method involves computing point clouds in both motions using the locations of the shoulders and spine of the character. Before computing the orientation from the closed form solution using the point clouds as input, the paper advises that we translate and rotate the point clouds so that the lowerback joint coincides with the origin. However, this seems to have introduced more twitching/noise (assuming I implemented correctly). I found that just making the root coincide with the origin to produce less noise, but I'm not entirely sure if that's the right approach. To troubleshoot whether I was doing things correctly, I wrote some code to visualize the point clouds in different colors for the two motions - this definitely helped to find quick orientation problems. The implementation involves conversions between Euler angles, rotation matrices, and quaternions. What initially caused me problems was the order of the Euler angles, which was YZX for the root and ZYX for the rest of the joints.

Results and observations coming soon...