Time Alignment

Another element in creating overlays is making sure both motions are in phase at the same time. For example, if you have two walking motions, you want both motions to be in the same phase of the walk cycle at each frame before doing a transplant.

Coincidentally, a lot of overlays already look as if they are in phase when doing a regular transplant. If we transplant the upper body of a character standing to the lower body of a character slowly walking, it does not look that bad. Problems however can occur when dealing with motions where the character is moving at different paces.

There were two methods I thought about to implement time alignment. One is dynamic time warping, which is used in the "Splicing Upper-Body Actions with Locomotion" paper by Heck, Kovar, and Gleicher. The other is uniform time scaling. I decided to go with dynamic time warping because I could also use it to check for similarity between motions (with a metric proposed in "Enriching a Motion Collection by Transplanting Limbs" by Ikemoto and Forsyth). The similarity checks was something I thought about to consider when auto generating overlays (I'll probably post more on that later).

The dynamic time warping approach computes point clouds for the lower bodies of both motions and creates a distance matrix, using an approach from the "Motion Graphs" paper by Kovar and Gleicher. I found that our framework had a similar metric for computing distances between frames so I decided to use that.

First, I had to make sure that the first frames of both motions were in phase. I compared the first frame of the shorter motion with frames of the longer motion to determine the most similar frame, and cropped the longer motion so both initial frames were similar enough.

I then looked at the dynamic programming method in "Flexible Automatic Motion Blending with Registration Curves" also by Kovar and Gleicher (at this point it seems like they do just about every relevant paper!) However, the paper didn't actually describe the algorithm so I had to look through a lot of other papers (many of which contained different elements that I didn't need) to find the correct method for our purposes. When computing the matrices, I also had to ensure that too many frames didn't map to a single frame (which guarantees a good slope in the time alignment curve). I set the limit so that there are at most 3 frames mapping to one. After computing the cost matrices, I computed the optimal path starting from the first frame and assigned each new frame based on the frame to frame mapping.

Here is an example of how this time alignment can improve an overlay:



Of course, time alignment is only useful when you have two motions that are moving similarly.

I may look into implementing uniform time alignment as well, if time permits. (To reiterate, this method was dynamic time warping.)

Spatial Alignment Results

After a lot of debugging and testing, I believe I have spatial alignment implemented correctly. I would say the difference between using spatial alignment and not using it is noticeable. In general though, the spatial alignment method can preserve details of the lower body motion before the transplant. But is that necessarily a good thing? Here are some of the results:



We can choose to try to align the upper body motion with the lower body motion (as done in the video above) or vice versa. From my observations, choosing either or does not improve all overlays and which is better depends on the overall postures in both motions and what the particular goal is. I'm assuming in most cases we want the upper body motion to adapt to whatever the lower body is doing so I'm leaving it as is for now. I'll have to look into possible ways to fix what is happening above.

I wanted to implement spatial alignment to solve the twitching problem caused because of the lack of correlation between the upper and lower body. Although it doesn't solve that problem in all cases, here's an example where spatial alignment can alleviate the twitching problem:



Notice that the alignment eliminates the unnecessary lean to the left in the original overlay. However, noise (in the form of very rapid twitching) is prevalent in a lot of the overlays. Perhaps we'll have to look into methods in filtering out the noise to produce better results.

Spatial alignment does not correlate the lower body with the new upper body in all cases. We have a lot of upper body motions that involve a character doing an action while standing. Even with alignment, these upper body motions look awkward when transplanted to a character that's running.

I'm currently working on time aligning the motions before transplanting limbs, although that presents some issues as we may be combining motions where the character is standing, running, or doing some other type of motion. I'm also looking into research for quantifying the naturalness of a motion, which could be useful when generating overlays. Lastly, I'm working on implementing the game environment in OGRE to visualize the overlays on a skinned character.

Implementing Spatial Alignment

First, let me recap why spatial alignment is important: when we transplant the upper body from one motion to the lower body of another, it is likely there will be noticeable problems in the form of twitching unless both motions were similar enough. This is because the upper body is usually correlated with lower body movements, and transplanting destroys this correlation.

Earlier in my project, I found a quick way to get rid of the twitching was to not transplant the joint above the root, the lowerback joint, which is responsible for rotation about the torso. Although excluding the lowerback joint eliminates the twitching, it sacrifices some rotation. A solution to this is proposed in "Splicing Upper-Body Actions with Locomotion" by Heck, Kovar, Gleicher [2006].

Following their method, called spatial alignment, we should compute a rotation for the lowerback joint that best aligns the shoulders and spine of the motion we use for the upper body with the motion for the lower body. The orientation is found by computing point clouds and solving for the closed form solution in "Closed-form solution of absolute orientation using unit quaternions" [Horn 1987].

Although it sounds simple enough, I've encountered a number of issues while trying to implement it. The closed form solution needs to compute eigenvalues of a system, so I had to find the appropriate matrix library to do that. I then implemented the solution and had to run many unit tests to ensure that it was implemented correctly.

The method involves computing point clouds in both motions using the locations of the shoulders and spine of the character. Before computing the orientation from the closed form solution using the point clouds as input, the paper advises that we translate and rotate the point clouds so that the lowerback joint coincides with the origin. However, this seems to have introduced more twitching/noise (assuming I implemented correctly). I found that just making the root coincide with the origin to produce less noise, but I'm not entirely sure if that's the right approach. To troubleshoot whether I was doing things correctly, I wrote some code to visualize the point clouds in different colors for the two motions - this definitely helped to find quick orientation problems. The implementation involves conversions between Euler angles, rotation matrices, and quaternions. What initially caused me problems was the order of the Euler angles, which was YZX for the root and ZYX for the rest of the joints.

Results and observations coming soon...

Creating a Game Environment and Shoulder Alignment

While we work on trying to fine tune the physics checks, I also have to think about other ways to improve the overlay motions...

My primary goals right now are:

1. Create the game environment in OGRE


2. Implement a method of shoulder alignment using point clouds to improve the overlays, which is described in "Splicing Upper-Body Actions with Locomotion" by Heck, Kovar, and Gleicher (I'm sure I mentioned this in an older post as I've been trying to find time to do this for a while).

I am hoping to get a preliminary version of the game up soon. Once that is done and assuming things go fairly smoothly, I'll be able to see how the overlays motions I generated look on another character model.

Generating Overlays using Physics Checks

I previously wrote a script to automatically generate overlays given the directories of motions desired for the lower body and upper body actions. The next step was implementing constraints to filter the results.

I integrated our physics checker (thanks to Aline) with the generation script to denote whether or not a created overlay is physically correct. Of course, the checker is not perfect, and even if it were, physically correct overlays are not guaranteed to look realistic or natural. For now, we're using it as one step in the right direction for generating better overlays.

With this script, I can generate a lot of different overlays. Currently, I'm examining which overlays I think look good (does it look like something a human would realistically do?) and why some others do not look as good.

Transplanting Individual Limbs

I improved the overlay creation tool to allow the user to transplant individual limbs. By default, all the limbs above the root will be transplanted. However, the user can select and de-select individual limbs. The interface is also very easy to use, allowing you to toggle between transplanting an entire arm. To demonstrate, here is a more recent video of the overlay creation process.



The user first loads the motion he/she wants to use for the lower body. Then the user selects to create an overlay and loads the motion for the upper body. Options include interpolating between the two motions before transplanting and selecting individual limbs as mentioned before.

Here is an example of the difference between an overlay where all the upper body limbs are transplanted and one where one of the arms is not transplanted:



In the example above, I am obviously using a running motion for the lower body and a drinking motion for the upper body. In the first overlay, all upper body limbs are transplanted from the drinking motion to the running motion. As a result, the right arm remains stiff because it wasn't moving in the original drinking motion. However, if we choose not transplant this arm we get the result shown in the second overlay, which I think looks a bit better.

Transplanting individual limbs can really help make better looking overlays. Of course, it also multiplies the number of possibilities we have to consider when thinking of ways to automatically generate overlays.

A Script to Generate Overlays

I previously implemented the ability to generate multiple overlays through the GUI. I decided to also create a standalone script to generate the overlays and parse through multiple directories of lower and upper body motions. This should be useful and quicker than having to rely on doing it through the motion viewer program.

Meanwhile, I've been studying previous works in closer detail. I will probably post a detailed review of related papers as the project continues.