Screenshots
Screenshots are on group webpage.
Document table of contents
A while ago, I had dinner at Cheeseburger in Paradise. I was looking at the cocktails menu. I noticed that most dirnks come with a small creature made of few fruits and wearing dark black sunglasses. So I thought about hiring some of these guys for our project 3. Our setting is a small caribbean island in the middle of the ocean where the characters are playing Jimmy Buffet's famour Cheeseburger in Paradise.
Storyboards
To kickoff the project, I made few storyboards illustrating my concept. The idea was to have three different characters made from a combination of fruits (including pineapple, strawberry, and lemon):
 Rum Runner |
 Electric Lizard |
 Pineapple creature |
 |
 |
 |
Initially, our plan was to directly import Vicon's V files directly into Ogre (using James' routines) for rendering. However, we later realized that simply rendering the raw Vicon files won't produce the desired affects, mainly due to noise in the datasets and difference in proportions between us (humans) and the virtual characters. We decided to use MotionBuilder to do re-targetting and some key-framing. However, my attempts to utilize MotionBuilder to cleanup the guitar player failed to produce a better result. I went back to Vicon's IQ system and used all possible cleanup features. The result on the guitar was acceptable (more on this later).
I made two characters: Rum runner and Lizard. I used Blender to create the models. For the texture, I used some textures from the web, and applied some filters to them using Photoshop so they look a bit more interesting. To model these creatures I usually start with a simple mesh (a cube, a circle, etc...) and extrude it in various direction. For the Lizard, I started with a head the looks like a slice of lemon I took from Kaushal and created upper body, hands, legs mainly using extrusion. To facilitate rigging, and to easily animate with the mocap data, all characters were modelled taking the famous T-Pos.
 |
 |
 |
Percussion set |
 |
Hay umbrella |
Initially, I created a skeleton and rigged the RumRunner and exported it to Ogre's mesh format so we could try to animate
it using some sample mocap datasets.
To our surprise, the results were slightly disappointing. Although the rigging process was done carefully, James' code was ripping the character apart!
|
|
| Initial attempt to animate character in MotionViewer | |
Kaushal was in charge of setting up the surrounding environment (ocean, island, etc...) in Ogre. My programming assignments were ensuring that the datasets load correctly, writing some code to synchronize the datasets as well as the audio, loading and playing the music file, and writing a simple configuration file parser so we could specify the physical location of band members along with their sources motion files, meshes, etc...
Dataset loading
As stated earlier, we're using James' routines to load Vicon V files that contain our mocap data and animate our characters. The code base we got from James proved very dependable, and it had very good Object-Oriented structure, which made the job of extending and using it simpler. However, no one has ever tried it on a PowerPC based machine (older generation Macs). When I first tried it, it just freezes after loading a Vicon file. After some troubleshooting, I discovered that the code implicity assumes an Intel architecture and loads all marker and bone orientation fields from the binary file directly to memory. Vicon happened to be using Intel's byte ordering for their file so this worked on Intel Macs, but rendered the data garbage on other architectures. So I had to write some simple routines to convert the bytes ordering of various fields in the Vicon file to PowerPC ordering.
Music playback
I used OpenAL to play the soundtrack file.
Dataset and Audio synchronization
The mocap datasets for each character was recorder individually. Therefore, all of them has to be synchronized with each others, as well as with the audio. To accomplish this, I added some routines to James' code that allow us to fast-forward/backward a certain number of seconds. To determine the suitable offest for each dataset, I kept adding/subtracting small offsets and trying them out until I was satisfies with synchronization.
Configuration file
A simple configuration file was created to allow pfeople to easily specify paths to motion files, as well as position of characters and temporal offsets for each mocap dataset for synchronization. This also proved valuable for testing the offsets without restarting the program (Press '.' to re-read config file options including temporal offsets and replay the music and animation from beginning).
I was the guitarist. We took 2 mocap sessions for the guitar. I used the typical full body mocap suit. For the guitar, we used the wooden stick that James made to capture the guitar as a seperate prop. The physical wooden prop looks like this:
A simple Vicon subject consiting of one bone was created to represent the guitar. The markers we arranged non-lineary to help the system track the bone.
The resulting datasets suffered from some noise. I applied several smoothing and gap filling functions decrease the noise. The processed is described in the following section.
A mocap dataset will never be perfect. An infrared-based motion capture system suffers from occluded markers, as well as limitation in identifying markers. This especially affects markers that remain occluded for a while and suddenly reappear, confusing the system, often causing it to mistakenly label that marker, and thus calculting an invalid orientation/translation for some bones. Luckily, there are a number of manual and automated cleanup and post-processing procedures that can be performed using Vicon's IQ software.
I used Vicon IQ to cleanup the percussion, guitar, and keyboards datasets. The dataset that required most work was the percussion, which took intensive manual cleaningup, as well as some automated filtering. The other datasets where not as bad, and an automated cleanup process was sufficient. The guitar dataset could have certainly used some additional manual cleaning up, but unfortunately, we ran out of time and couldn't perform all cleanups that we wanted. Below I will describe the cleaning up for the percussion. For the other two datasets, I used a subset of this process. The process goes as follows:
Marker labeling
Typically, one has to do marker labeling once only after capturing the ROM (range-of-motion) dataset. In practice, a lot of markers go unlabelled by the system, and many more gets mistakenly labelled. An unlabelled marker reduces the accuracy of bone data (orientation/translation) for one or more bones, whereas a mistakenly labelled one could easily screw up several seconds of the mocap data. Unfortuantely, this process has to be done manually. To looks for unlabelled or mistakenly labelled markers, one has to play the mocap dataset in the Vicon IQ software at a slow rate (I used 15-20% of actual speed), and look for unlabelled (white) markers. Alternatively, I sometimes turn on bone viewing, and look for segments where there is an obvious hitch or some ackward bone rotation. Then I thorougly inspect the frames surrounding this sections looking at the markers. If I find unlabelled markers, I label them. If I find some mistakenly labelled markers (one can tell by looking at surrounding markers or looking at marking of previous frames), I also correct them. The correction/labelling is simply done by selecing the correct name from the list of markers, and clicking the desired marker to be labelled/corrected.
Manaul labelling process
The process is unfortunately is extremely time consuming. Examining and relabeling the markers of the percussion dataset took me more then 12 hours to correct only 2:30 minutes of mocap data! However, this is a very good opportunity to fix up a lot of the mess left behind bad markers :)
Gap Filling
Gaps occur when the system can't see one or more marker. There's nothing much we can do here; missing information can not be recovered with full certainty. The best we can hope for is guessing the position of missing markers by interpolating their position across the gaps. There are 3 types of gap filling available in Vicon IQ: Spline, Linear, and Kinematic model based. The Spline and Linear models work only on small gaps (10-20 frames when mocap rate is 120 frames/sec). Anything more causes a mess! When I started using the post-processing system, I was setting the maximum gap for spline gap filling to around 70 frames. I didn't know that this was causing extremely bad resutls, until I re-exmined the mocap data after the filling. And it was a real mess. Gap filling using kinematic model is supposed to work well with large gaps (as the default maximum gap length is set to 1,000,000 frames). It works after the Kinematic Fitting operation. It produces a smooth, realistic interpolation, but again, this might not be the actual movement of the person, but its better than Linear and Spline gap filling for larger gaps.
The lower portion of the workspace displays marker data for the selected marker.
Sections hilighted in red indicate gaps that have ben filled
Kinematic Smoothing
After I labelled as much markers as I could and filled all the gaps, a Kinematic Smoothing operation is useful for removing hitches. This automated process loops through all the frames and applies some filters at the bone level to interpolate and smooth rotation/translation of bones. The result is a smoother motion with less visible hitches. This process is computationally intensive. It took about 1 hour to smooth 2:30 minutes of motion (for each dataset).
MotionBuilder
After capturing the guitar data, we realized that it will not simply fit into the character because of difference in proportions. The guitar would occasionally go into the body of the character and the hands are not well aligned with the guitar. We decided to use MotionBuilder to do re-targetting and some key-framing.
To export motion datasets into MotinoBuilder, I exported the mocap of the guitar to 2 files: one containing my motion, and the other one containing the guitar's motion. I imported the Vicon V file into MotionBuilder using Vicon's plugin. It took roughly about 2 hours to complete the import process. After that I "characterized" my motion dataset, and saved it to a seprate FBX (MotionBuilder's native format) file. I started a new scene, imported the character's and the guitar's meshes to MotionBuilder (Anjul converted the meshes from Maya to FBX), and also opened both motion files in one scene. I told the character to use the motion dataset as source of motion, and parented the guitar's bone to the guitar motion.
Meshes and datastes imported into MotionBuilder
I used the "global" Inverse Kinematic keys to key-frame certain bone orientation/translation for the hands of the character so that they look like they're playing the guitar. The nice thing about these keys is that they are applied as a seperate layer over the original motion, and they affect the whole motion dataset. Additionally, MotionBuilder includes functionalities for automatic re-targetting of certain portions of the characters (such as the feet).
Retargetting and Keying Inverse Kinematic
Unfortunately, I was running out of time, and since I didn't have previous experience with MotionBuilder, I couldn't get the desired results out of it. I must say that James gave us a second workshop about MotionBuilder, but I simply couldn't get it right :( We decided to completely abandon MotionBuilder, and utilize Vicon's IQ software for cleaning up. Eventaully, Anjul was able to sculpture the guitarists so that the guitar doesn't go inside it very often. This significantly imporved the overall result, but there are still places where its obvious that the guitar goes inside the body. The left hand also didn't look very realistic. But, I guess that's the best we could get without a tool for retargetting and key-framing.
This has been a very interesting expierience. I learned how to use a professional mocap system, what the limitations are, and what needs to be done after the mocap data has been acquired. It is really important to spend a reasonable time on each step of the pipeline. Getting a good quality data requires attention to minute details including careful marker placement, thorough ROM (range-of-motion) process, and finally manual and automatic cleaning up of the data.