Encoding Fraps videos: solving colorspace issues (wrong color, faded, etc)

This is a long post, sorry about that! Its purpose is to provide some encoder settings to preserve color fidelity. The correct settings to use depend on the source, the application doing the encoding, and the application you want to play the video. Reading the background stuff is optional. If your encoding application is listed here, you can use the suggested settings. If you aren't sure how this works, or if your encoding application is not listed, post a reply and I will help you with it.
What color space is Fraps? *
What settings should you use to encode with, if you want correct colors? Why do we sometimes get faded or exaggerated colors with StaxRip, or VirtualDub, or Vegas, or Premiere or other encoding programs? Over the past couple of weeks, I did a lot of testing to try to answer these questions.

The first problem is, what is normal? What are the colors supposed to be? Can we know what the game designer intended? There are so many variables that can affect the color on a game - game settings, video card settings, even driver versions. I had to start with known colors, and to get that, I cobbled together the Fraps Test Pattern program. The program simply puts an image on the screen in a way that Fraps can record. It's made as simply as possible, both because I don't know much about DirectX programming, and because I didn't want the added features (especially lighting) of a more 'advanced' program to affect the color of my source image. Now I could put an image of my choice on the screen and record it with Fraps (or any other DirectX capture program).

Playing the video on most (not all!) players showed there was no color shift between the original image and the Fraps video. Great! (I wasn't sure that was going to be true at all). I always check the colors on screen with the excellent ColorPic program, to confirm my eyeball evaluation. Then I started compressing the Fraps videos with different encoders to compare the quality. I was shocked when I compressed a video with my favorite program, VirtualDub, with x264vfw (H.264), uploaded the result to YouTube, and found the reds were exaggerated and the greens were weak. What - how can this be!? Well, that started a marathon testing session of every encoder and video editor I had available, and guess what - there were problems like that everywhere I looked. The good news is, if you know what's happening, you can easily correct for it - most of the time.

First, what kind of errors are we talking about? What do they look like?
Here's a cropped version of my test image:


Here's the image with "red push;" over-saturated reds and weak greens:


Here's the image with "green push;" over-saturated greens and weak reds:


The color errors are are subtle with normal, not-very-colorful source material, but they're easy to spot with color bars - or any highly saturated feature with known color.

What causes these errors? I can't pretend to know why exactly for every situation, but in general, "green push" is due to a video which has been converted to YUV (and sent to the encoder) in the Rec.709 (or HD) color space, but converted back to sRGB (and sent to the monitor) in the Rec.601 (or SD) color space. "Red push" is due to the reverse: stored as Rec.601, displayed as Rec.709. I think. The internal machinations don't matter too much, for me; the important thing is to check for errors at each stage of the signal chain and correct for them. I will explain how that's done down-thread.

You can get away with a color space error, and most people won't notice (some of you do, because you've been posting about it: here, here and here); however, if you get multiple color errors in your signal chain (eg, boost the reds, then re-encode and boost them again), the video quality will fall apart very quickly. The results of my tests should be helpful, but if you use any tools or settings I haven't listed below, you need to test for yourself.

Luma range errors were seen in my testing as well. More on luma range errors on the "Fraps Dark" thread, here. They need to be caught and corrected at each stage of the signal chain too.

To review, here's the test image with "blowout" or exaggerated luma range:
(sometimes called "too dark", but it's also "too bright" - detail is lost at both ends of the scale)


And here is the image with "washout" or low luma range:


* What color space is Fraps? Short answer, Rec.709, full range; but in practice, the color space you see is whatever the decoder spits out. If you use the Fraps decoder, the output is RGB, full range. Other decoders may return something else. That's why it's important to test.

 

In all my tests, I was thinking about two main scenarios: one, uploading to YouTube, and two, encoding losslessly for further editing. Lossless encoders aren't handled well by YouTube (and the upload takes too long), and video compressed for YouTube isn't handled well by video editors (random seek takes forever).

The programs and codecs I used for encoding were:

• ffmpeg (command line)
  • codecs: mjpeg, mpeg4, libx264, libx264rgb, huffyuv
• "VDB" (VirtualDub 1.10.2)
  • codecs: ffdshow/mjpeg, divx, xvid, mpeg4, wmv9, x264, huffyuv, lagarith
• Avidemux 2.5
  • codec: x264
• Handbrake
  • codec: x264
• "MS-Expr" (Microsoft Expression Encoder)
  • codec: wmv9
• "PE4" (Premiere Elements 4)
  • codec: x264vfw
• "PE4->VDB" (Premiere Elements 4, frameserving to VirtualDub)
  • codec: x264vfw
• StaxRip
  • codec: x264

The programs I used for playback evaluation were:

• "VDB" (VirtualDub)
• "PE4" (Premiere Elements 4)
• "MP:H" (Media Player Classic Home Cinema, with output to "Haali")
• "MP:R" (MPC-HC with output to "VMR9 renderless")
• "MP:W" (MPC-HC with output to "VMR9 window")
• "BS:O" (BSPlayer 1.02 with output to "overlay")
• "SM:G" (SMPlayer 0.8.0 with output to "gl-fast")
• "FF" (ffplay.exe command line player, part of ffmpeg)
• "YT" (YouTube)

And of course, I had to test Fraps in RGB mode and in normal (YUV) mode...

There were a heck of a lot of combinations to test. Fortunately, a pattern emerged and I was able to skip some of the tests because I knew that if program "A" did something, program "B" would do the same. Still, it was a lot of work - especially since I started over a couple of times when I found errors in my methods.

continued in next post...

 

I'll try to keep this as short as possible, but that's a hard thing to do with this much raw data. If anything's not clear, please ask; I've been looking at this so long, it's hard for me to see it objectively. I'll try laying it out like this:

• <application>
  • codec: <codec name>
    • (notes)
    • subheading: Fraps YUV or RGB source video
      • list of players (see abbreviations, above) "=" color fidelity evaluation
      • list of players "," workarounds (encoder settings to get proper playback, if known)

• ffmpeg
  • codec: mjpeg
    • Fraps YUV source video; output viewed with...
      • YouTube and most media players displayed correct colors (= ok)
      • VDB and PE4 = blown-out (y+)
      • SM:G and FF = green push (g+)
        I didn't look into workarounds; maybe later
  • codec: mpeg4 (MPEG-4 ASP)
    • always add "-pix_fmt yuv420p"
    • Fraps YUV source video; output viewed with...
      • SM:G and FF, add "-vf colormatrix=bt709:bt601"
      • all others = ok
    • Fraps RGB source video; output viewed with...
      • SM:G, FF = ok
      • all others, add "-vf colormatrix=bt601:bt709"
  • codec: libx264
    • always add "-pix_fmt yuv420p"
    • Fraps YUV source video; output viewed with...
      • VDB, PE4, SM:G, FF, add "-vf colormatrix=bt709:bt601"
        BTW, "-colorspace 6" would have been slightly better, since
        "-vf colormatrix" changes the video data, but "-colorspace" only
        adds a tag; but unfortunately, "-colorspace" seemed to have no effect.
      • all others = ok
    • Fraps RGB source video; output viewed with...
      • VDB, PE4, SM:G, FF = barely perceptible red push
      • all others = r+
        I didn't look into workarounds; maybe later; I think would
        write an Avisynth script to bring Fraps RGB into ffmpeg
  • codec: libx264rgb
    • Fraps RGB source video; output viewed with...
      • PE4 = r+
      • MP:H, YT = error (scrambled video)
      • all others = ok
  • codec: huffyuv (lossless)
    • Fraps YUV source video; output viewed with...
      • VDB, PE4 = ok
      • MP:H, MP:R, MP:W, YT, add "-vf colormatrix=bt709:bt601"
    • Fraps RGB source video; output viewed with...
      • VDB[1], PE4, MP:H, MP:R, MP:W = ok (add "-pix_fmt rgb24")
      • others not tested

continued in next post...

 

• "VDB" (VirtualDub 1.10.2)
  • add the Alias Format filter (set color space = Rec.709; range = no change).
  • codec: ffdshow/mjpeg
    • (required a complicated setup for mediocre results)
  • codec: DiVX; XViD; Microsoft MPEG-4 ASP; MS VC-1
    • Fraps YUV source video; output viewed with...
      • MP:H, BS:O = r+
      • others not tested
        didn't spend much time with these codecs
  • codec: x264vfw
    • Fraps YUV or RGB source video; output viewed with...
      • VDB, PE4, SM:G, FF = ok
      • all others, add "--colormatrix bt709"
  • codec: Huffyuv (lossless)
    • Fraps YUV source video (Huffyuv mode= YUY2); output viewed with...
      • VDB = ok
      • PE4 = freezes[2]
      • MP:H, MP:R, MP:W, YT = r+
        will explain workaround later maybe
    • with Fraps RGB source video, (Huffyuv mode= RGB)
      • VDB, MP:H, MP:R, MP:W = ok
      • others not tested
  • codec: Lagarith (lossless)
    • with Fraps YUV source video (Lagarith mode=YV12); output viewed with...
      • VDB, PE4, MP:H, MP:R, MP:W = ok
      • YT = error (upload failed)
      • others not tested
    • with Fraps RGB source video (Lagarith mode=RGB); output viewed with...
      • VDB, PE4, MP:H, MP:R, MP:W, BS:O, FF = ok
      • SM:G = error (crash, IIRC)

NOTES:
VDB[1]=must open with DirectShow plugin
freezes[2]=need to test again with PCM audio instead of MP3

continued in next post...

 

• Avidemux 2.5
  • codec: x264.exe
    • Fraps YUV source video; output viewed with...
      • VDB = y+ and g+
      • MP:H = y+
        could not find a workaround
• Handbrake
  • codec: x264.exe
    • Fraps YUV source video; output viewed with...
      • VDB[1], MP:H = ok
      • others not tested
• "MS-Expr" (Microsoft Expression Encoder)
  • codec: Microsoft WMV9 VC-1 Advanced
    • Fraps YUV source video; output viewed with...
      • VDB[1], PE4, SM:G, FF = ok
      • all others = r+
• "PE4" (Premiere Elements 4)
  • codec: x264vfw
    • Fraps YUV source video; output viewed with...
      • VDB = ok
      • MP:H, add "--colormatrix bt709"
      • others not tested
      • very high bitrate for some reason
• "PE4->VDB" (Premiere Elements 4, frameserving to VirtualDub)
  • codec: x264vfw
    • Fraps YUV source video; output viewed with...
      • VDB = ok
      • MP:H, add "--colormatrix bt709"
      • others not tested
• StaxRip
  • input script for Fraps sources: ConvertToYV12(matrix="Rec709") [see here]
  • codec: x264.exe
    • x264: set colormatrix=bt709 (did not seem to have any effect)
    • Fraps YUV or RGB source video; output viewed with...
      • VDB, MP:H, YT, BS:O = ok
      • MP:W = y-
      • SM:G = g+

NOTE: Any time MP:H is listed but YT is not, that's because YT always (with one exception) followed MP:H. The exception was a test video uploaded at standard def (848x480), which was ok on MP:H but r+ on YT.

continued in next post...

 

An interesting post on doom9.org by Petrakeas2, "BT.601 and BT.709 compatibility benchmark," confirms most of my tests, and my tests confirm most of his.

I will try to sum it all up here, if possible...

There are six ways to handle color space, given YUV input. RGB input (such as produced by the Fraps decoder) is not affected.

1: Always use Rec.601 (the old standard-definition color space); if given Rec.709, display it with green push.​

​

2: Always use Rec.709 (the high-definition color space); if given Rec.601, display it with red push.​

​

3: Use Rec.709 for HD only: if resolution is higher than a certain threshold (approximately 480p), assume Rec.709 (display Rec.601 as red); else assume Rec.601 (display Rec.709 as green).​

​

1a, 2a, 3a: Whether using strategy 1, 2 or 3, use color-space tags in the video stream, if available, to decide. If the tags are correct, great. If not, the colors will be wrong.​

I didn't test methodically to see which strategy each player was using; for one thing, I was only interested in high definition - 720p or above, although I did run one quick test at 480p. Therefore I can't tell (and am not too interested in) the difference between strategy 3 (Rec.709 for HD only) and strategy 2 (always use Rec.709). Petrakeas2 goes into SD behavior in more detail.

Here's how I see the behavior of the applications I tested:

• VirtualDub always assumes Rec.601, unless the Rec.709 tag is added (x264 "--colormatrix bt709"). Otherwise, Rec.709 input (g+) can be corrected with the Alias Format filter (set color space = Rec.709; range = no change). To see the correction in still-frame, at least one other color filter must be active. I suggest the HSV filter, left at default settings.
  .
• Premiere Elements 4 always uses Rec.601. Not aware of any way to correct it. Not sure how newer versions work, but see the note about Adobe CS5.5 below.
  .
• Media players using the Haali renderer seems to use Rec.709 for HD only. (I find the choice of video renderer matters more than the media player application itself)
  .
• Media players using the VMR9 renderer, whether the "window" flavor or the "renderless" flavor, seem to use Rec.709 for HD only.
  .
• Media players using the Overlay option seem to use Rec.709 for HD only, but they display reduced luma range (washout) with WMV9/V-1 in my tests; also with StaxRip/x264 output, but only with Fraps RGB sources.
  .
• YouTube seems to assume Rec.709 always...
  • Always Rec.709 in HD for me; Rec.709 in SD based on only one test (with x264)
  • Note this can vary depending your hardware acceleration settings, and possibly other factors. My findings here disagree with Petrakeas2's, just so you know. I am using WinXP and a NVidia DX9 GPU; Flash player has hardware acceleration enabled.
  • More testing, on a variety of systems, is needed. You can look at my test videos on YouTube and tell me if you see what I am seeing.
  • I agree with Petrakeas2 that having multiple Flash windows open will affect the color; only the first window will have hardware acceleration.
  • If my tests are valid, using a media player with the Haali renderer is the best way to "preview" the color as it will appear on YouTube, assuming 720p resolution or more.

According to Petrakeas2 (and I believe him),

• Adobe Suite CS5 always use Rec.601, but CS5.5 uses Rec.709 for HD only.
  .
• Sony Vegas uses Rec.709 for HD only. I believe there are filters to correct this if needed -- anyone know?

 

(place holder)

 

Bump - finished!
Feedback is welcome. Your questions will help me make this more clear, I hope - and more correct, if I'm wrong somewhere.

Again, if you aren't sure how this works, or if your encoding application is not listed, post a reply and I will help you with it.

- - - - - - - - - - - - - - - - - - - - - - -
EDIT Aug 8, 2012 - nothing huh? I guess you see a wall of text and step away. I don't blame you.
I blame myself for my unreadable style. I'll try again:
https://encodingtalk.com/threads/fraps-colorspace-issues-wrong-color-faded-dark-etc-part-2.1613/

 

The Fraps codec always decodes as RGB, whether the recording was made in RGB or YUV.

Fraps YUV format is full range, Rec.709. Source: Fraps developer via doom9.

I had to write my own DirectX test pattern generators to confirm this. Here is my latest version: Frafs Test Pattern 0.5

FFmpegSource2 (requires Avisynth) decodes Fraps as YUV or RGB as required; this is a better way to go, as the Fraps color resampling (YUV to RGB) is a bit soft. With access to the original YUV, you can apply your own chroma resampling method when you convert to RGB.

If you see colors wrong when decoding as full range 709, it's for one of three reasons, that I know about.

1. You have not calibrated your playback signal chain. I address that and more in Fraps colorspace issues (wrong color, faded, dark etc) - part 2, and before that, in the Fraps 'dark' thread, post #40 and post #52.
   .
2. You have color management software running which needs to be disabled during capture.
   .
3. The game is doing gamma manipulations post frame buffer. You can try adjusting "brightness" in the game, and/or you must color correct in post.

 

>It will take that image, convert it to YV12 With Rec.601
Why would you do that :/

Code:

FFmpegSource2("video.avi", atrack=1)
## OPTIONAL: convert to RGB only if required by downstream filters etc
#ConvertToRGB32(matrix="PC.709", chromaresample="spline16")

This assumes the signal destination handles full range 709 correctly. For example, VirtualDub expects Rec.601, so you must use the Alias Format filter; Color=Rec.709, Range=No Change.

>you can´t make rec.709 to rec.601 and vice versa, losslessly
That is absolutely correct. So try to convert as seldom as possible; don't go back and forth. Same for luma range and for YUV<>RGB. Eventually, it will probably have to be YUV, studio range in order to be encoded and viewed (ignoring 4:4:4 for now), but when this conversion is done depends mainly on your video editor. What does it expect to see? Trying to answer that question was the genesis of this thread.

> i don´t think anyone really cares
I've kind of made it my business to care, but hopefully, not to an obsessive degree

 

I have Some of my test images - compare:



(Fraps YUV ... FFMpegSource ... Fraps RGB)
FFMpegSource is very slightly sharper; RGB is a whole lot sharper.

 

Hi Bregor, either way is OK, but I would add the x264 command --colormatrix bt709, or equivalent GUI option if using StaxRip etc; this is not always needed (depending on the media player you are using, since many assume Rec709/BT709 by default), but it never hurts.

Instead of .ColorYUV(levels="PC->TV"), try the VirtualDub Alias Format Filter, range=full instead; the quality is a little better.

All in all, I prefer FFVideoSource/FFAudioSource, as the colors are a wee bit sharper; plus ConvertToYV12 is not needed.

Regarding VirtualDub: VirtualDub always previews YUV assuming Rec609 - so greens look a more intense, and reds a little less, than they will on the final video. To fix this, add an Alias Format Filter, color space=Rec.709 - for previewing purposes ONLY. Disable before encoding. This does not apply to previewing RGB video.

 

It's a little very confusing. I went and consulted the source code, so I'm pretty sure this is correct:

matrix="Rec709" assumes RGB is 0-255 and YUV is 16-235;

so full-range RGB->ConvertToYV12(matrix="Rec709")

gives TV-range YV12,​

and TV-range YV12->ConvertToRGB32(matrix="Rec709")

gives full-range RGB.​

This is usually what you want. In the case of the Fraps codec,
which is full-range RGB, the correct conversion to TV-range YV12 is performed.
Rec601 works the same way, but the colors are slightly different.

matrix="PC.709" keeps the same luma range, which is not used very often.
It might be useful for converting Fraps+FFVideoSource full-range YV12 to RGB
without exaggerating the luma range.

BTW for YUV<->YUV and RGB<->RGB conversions, matrix has no effect.

EDIT: tested and verified.

 

No problem, there are probably other people with the same questions. You are very close: