Keying Difficult Footage

Sometimes you are given source footage that is just tremendously hard to key. Primatte Keyer Pro has a host of tools to deal with compression, spill and color correction but sometimes noise and lack of dynamic range can cause holes to appear in the matte or maybe an edge that looks like it fluttering, ruining the composite effect.

All the steps below use After Effects as the program, while some of the functions covered can be done in other tools, After Effects is the best tool to work with tough footage.

Approaches

There are four main approaches that we will discuss to help salvage footage that you may have difficulty keying:

• Smoothing color channels

• Pre-keying color correction

• Garbage matte prior to using Primatte Keyer Pro

• Proper sampling in primatte to avoid edge problems

Smoothing Color Channels

Primatte Keyer Pro 4.0 features a deartifacting pass that will create nice sharp edges even with sub-sampled footage from DV, HDV or MPEG-2 source formats. Sometimes though this deartifacting while preserving edges does not generate a great key because the color information in the image has lots of noise.

Here is an example of a poorly lit green screen example that needs some smoothing to help eliminate some noise in the color channels.

So you might notice that the there is a large seam in the background and that there is a lot of blocky noise artifacts in the green screen. Let's tackle the green screen smoothing first.

Here is the Red Channel from this same image, so you can more easily see the blocking and the noise.

Notice the aliasing along the shoulder and neck and the large blocks in the background.

With After Effects, you can easily smooth just the color channels by using a Channel Combiner effect (a terrible name) and a simple blur. This will allow you to smooth just the chroma channels in the image.

The steps are Apply Channel Combiner set to RGB to YUV, Apply a Channel Blur with Blue and Green channels set 2.0 pixel values and then use the Channel Combiner again set to YUV to RGB. This will essentially smooth just the chroma channels an reduce the noise making for a nicer looking matte. This is the right settings for HDV or footage captures with a 4:2:2 codec. If you are using DV footage then set the Channel Blur to 3.0 instead of 2.0.

Here is the plug-ins applied in After Effects Effect Control window.

Finally, here is the smoothed result after using these plug-ins below.

Full credit to Mark Christiansen author of After Effects CS3 Studio Techniques for this tip.

Color Correction for Good Keys

The color values in the footage have a big impact on the final keyed output. Footage that is too dark or lack enough separation between the foreground colors and the background can give really bad results.

Here is the uncorrected shot again (below).

Now clearly this is very dark. Let's apply some exposure correction and Hue/Saturation changes with Magic Bullet Colorista and the After Effects Hue/Saturation tool.

As you can tell from the interface the Gain has been pushed to blue and the exposure brought up by 2 stops. The Hue and Saturation controls are set to simply boost the green and make the green more pure. It useful to do these changes with the project set to 32bit mode so that the color correction has the greatest precision.

Here is the result (below).

Garbage Mattes

The footage we started with below has noticeable dark corners. Using a simply garbage mask will let you remove those edges. A garbage mask is any shape that you animate to cut out parts of the footage. It is called "garbage" because you are cutting out the garbage and not trying to create a perfect fitting mask.

Removing the edges has the added benefit of not having to include those colors in the key that Primatte Keyer builds. By leaving those colors out, Primatte Keyer generates a more accurate matte. You don't have to create tight outlines, just move a couple of shapes around the subject. Animate by creating keyframes every 10-12 frames. Don't worry about animating points, just move shapes. You should be able to matte the footage in just a few minutes in After Effects.

The complete footage is shown below. Notice the darker edges and folds in the outside edges of the screen area? Cutting these out manually will make for a better key.

Here is the version with simply garbage mattes (below).

Using two shapes makes quicker work because you only have to move the shapes and not the individual points on the garbage mask shapes themselves. I was able to create outlines for 30 seconds of green screen footage in about 10 minutes. If you use this footage as input source, the alpha channel created by After Effects mask will simply add to the matte that Primatte Keyer creates.

Proper Sampling

With this footage in particular, there is a bright greenish/yellow spot on the card fixed to the hat. If you try to include this in the key area, you will end up creating a key with too much edge spill that will look rough and unconvincing. Here is a case where if you you leave this sample out you can just fix it with Alpha Cleaner and get a better key.

Here is a mostly finished key of our guy but notice the dark area in the hat. This will appear as a big hole in matte.

So if we turn on Alpha Cleaner with the Plug Holes mode and do some control tweaking here is the result below.

Here are the settings we used to remove the big specks but keep the edge untouched and even retain the fine transparency on the left side of the image, which is really important to keep the hair in the shot looking realistic.

Final Results

Of course any image can look good on a single frame, but trust me when I say that using these approaches and enabling the smoothing, alpha cleaner and light wrap controls results in great looking output, even for low light shots.

Go from this (before):

To this finished composite: