Time Alignment

Another element in creating overlays is making sure both motions are in phase at the same time. For example, if you have two walking motions, you want both motions to be in the same phase of the walk cycle at each frame before doing a transplant.

Coincidentally, a lot of overlays already look as if they are in phase when doing a regular transplant. If we transplant the upper body of a character standing to the lower body of a character slowly walking, it does not look that bad. Problems however can occur when dealing with motions where the character is moving at different paces.

There were two methods I thought about to implement time alignment. One is dynamic time warping, which is used in the "Splicing Upper-Body Actions with Locomotion" paper by Heck, Kovar, and Gleicher. The other is uniform time scaling. I decided to go with dynamic time warping because I could also use it to check for similarity between motions (with a metric proposed in "Enriching a Motion Collection by Transplanting Limbs" by Ikemoto and Forsyth). The similarity checks was something I thought about to consider when auto generating overlays (I'll probably post more on that later).

The dynamic time warping approach computes point clouds for the lower bodies of both motions and creates a distance matrix, using an approach from the "Motion Graphs" paper by Kovar and Gleicher. I found that our framework had a similar metric for computing distances between frames so I decided to use that.

First, I had to make sure that the first frames of both motions were in phase. I compared the first frame of the shorter motion with frames of the longer motion to determine the most similar frame, and cropped the longer motion so both initial frames were similar enough.

I then looked at the dynamic programming method in "Flexible Automatic Motion Blending with Registration Curves" also by Kovar and Gleicher (at this point it seems like they do just about every relevant paper!) However, the paper didn't actually describe the algorithm so I had to look through a lot of other papers (many of which contained different elements that I didn't need) to find the correct method for our purposes. When computing the matrices, I also had to ensure that too many frames didn't map to a single frame (which guarantees a good slope in the time alignment curve). I set the limit so that there are at most 3 frames mapping to one. After computing the cost matrices, I computed the optimal path starting from the first frame and assigned each new frame based on the frame to frame mapping.

Here is an example of how this time alignment can improve an overlay:



Of course, time alignment is only useful when you have two motions that are moving similarly.

I may look into implementing uniform time alignment as well, if time permits. (To reiterate, this method was dynamic time warping.)