Encoding Fraps videos: solving colorspace issues (wrong color, faded, etc)

Discussion in 'FRAPS General Discussion, Guides, and Tutorials' started by raffriff, Jul 20, 2012.

  1. raffriff

    raffriff Moderator Staff Member Site Contributor

    This is a long post, sorry about that! Its purpose is to provide some encoder settings to preserve color fidelity. The correct settings to use depend on the source, the application doing the encoding, and the application you want to play the video. Reading the background stuff is optional. If your encoding application is listed here, you can use the suggested settings. If you aren't sure how this works, or if your encoding application is not listed, post a reply and I will help you with it.
    200px-CIExy1931_sRGB.svg.png What color space is Fraps? *
    What settings should you use to encode with, if you want correct colors? Why do we sometimes get faded or exaggerated colors with StaxRip, or VirtualDub, or Vegas, or Premiere or other encoding programs? Over the past couple of weeks, I did a lot of testing to try to answer these questions.

    The first problem is, what is normal? What are the colors supposed to be? Can we know what the game designer intended? There are so many variables that can affect the color on a game - game settings, video card settings, even driver versions. I had to start with known colors, and to get that, I cobbled together the Fraps Test Pattern program. The program simply puts an image on the screen in a way that Fraps can record. It's made as simply as possible, both because I don't know much about DirectX programming, and because I didn't want the added features (especially lighting) of a more 'advanced' program to affect the color of my source image. Now I could put an image of my choice on the screen and record it with Fraps (or any other DirectX capture program).

    Playing the video on most (not all!) players showed there was no color shift between the original image and the Fraps video. Great! (I wasn't sure that was going to be true at all). I always check the colors on screen with the excellent ColorPic program, to confirm my eyeball evaluation. Then I started compressing the Fraps videos with different encoders to compare the quality. I was shocked when I compressed a video with my favorite program, VirtualDub, with x264vfw (H.264), uploaded the result to YouTube, and found the reds were exaggerated and the greens were weak. What - how can this be!? Well, that started a marathon testing session of every encoder and video editor I had available, and guess what - there were problems like that everywhere I looked. The good news is, if you know what's happening, you can easily correct for it - most of the time.

    First, what kind of errors are we talking about? What do they look like?
    Here's a cropped version of my test image:

    Here's the image with "red push;" over-saturated reds and weak greens:

    Here's the image with "green push;" over-saturated greens and weak reds:

    The color errors are are subtle with normal, not-very-colorful source material, but they're easy to spot with color bars - or any highly saturated feature with known color.

    What causes these errors? I can't pretend to know why exactly for every situation, but in general, "green push" is due to a video which has been converted to YUV (and sent to the encoder) in the Rec.709 (or HD) color space, but converted back to sRGB (and sent to the monitor) in the Rec.601 (or SD) color space. "Red push" is due to the reverse: stored as Rec.601, displayed as Rec.709. I think. The internal machinations don't matter too much, for me; the important thing is to check for errors at each stage of the signal chain and correct for them. I will explain how that's done down-thread.

    You can get away with a color space error, and most people won't notice (some of you do, because you've been posting about it: here, here and here); however, if you get multiple color errors in your signal chain (eg, boost the reds, then re-encode and boost them again), the video quality will fall apart very quickly. The results of my tests should be helpful, but if you use any tools or settings I haven't listed below, you need to test for yourself.

    Luma range errors were seen in my testing as well. More on luma range errors on the "Fraps Dark" thread, here. They need to be caught and corrected at each stage of the signal chain too.

    To review, here's the test image with "blowout" or exaggerated luma range:
    (sometimes called "too dark", but it's also "too bright" - detail is lost at both ends of the scale)

    And here is the image with "washout" or low luma range:

    * What color space is Fraps? Short answer, Rec.709, full range; but in practice, the color space you see is whatever the decoder spits out. If you use the Fraps decoder, the output is RGB, full range. Other decoders may return something else. That's why it's important to test.
  2. raffriff

    raffriff Moderator Staff Member Site Contributor

    In all my tests, I was thinking about two main scenarios: one, uploading to YouTube, and two, encoding losslessly for further editing. Lossless encoders aren't handled well by YouTube (and the upload takes too long), and video compressed for YouTube isn't handled well by video editors (random seek takes forever).

    The programs and codecs I used for encoding were:
    • ffmpeg (command line)
      • codecs: mjpeg, mpeg4, libx264, libx264rgb, huffyuv
    • "VDB" (VirtualDub 1.10.2)
      • codecs: ffdshow/mjpeg, divx, xvid, mpeg4, wmv9, x264, huffyuv, lagarith
    • Avidemux 2.5
      • codec: x264
    • Handbrake
      • codec: x264
    • "MS-Expr" (Microsoft Expression Encoder)
      • codec: wmv9
    • "PE4" (Premiere Elements 4)
      • codec: x264vfw
    • "PE4->VDB" (Premiere Elements 4, frameserving to VirtualDub)
      • codec: x264vfw
    • StaxRip
      • codec: x264
    The programs I used for playback evaluation were:
    • "VDB" (VirtualDub)
    • "PE4" (Premiere Elements 4)
    • "MP:H" (Media Player Classic Home Cinema, with output to "Haali")
    • "MP:R" (MPC-HC with output to "VMR9 renderless")
    • "MP:W" (MPC-HC with output to "VMR9 window")
    • "BS:O" (BSPlayer 1.02 with output to "overlay")
    • "SM:G" (SMPlayer 0.8.0 with output to "gl-fast")
    • "FF" (ffplay.exe command line player, part of ffmpeg)
    • "YT" (YouTube)
    And of course, I had to test Fraps in RGB mode and in normal (YUV) mode...

    There were a heck of a lot of combinations to test. Fortunately, a pattern emerged and I was able to skip some of the tests because I knew that if program "A" did something, program "B" would do the same. Still, it was a lot of work - especially since I started over a couple of times when I found errors in my methods. :(

    continued in next post...
  3. raffriff

    raffriff Moderator Staff Member Site Contributor

    I'll try to keep this as short as possible, but that's a hard thing to do with this much raw data. If anything's not clear, please ask; I've been looking at this so long, it's hard for me to see it objectively. I'll try laying it out like this:
    • <application>
      • codec: <codec name>
        • (notes)
        • subheading: Fraps YUV or RGB source video
          • list of players (see abbreviations, above) "=" color fidelity evaluation
          • list of players "," workarounds (encoder settings to get proper playback, if known)
    • ffmpeg
      • codec: mjpeg
        • Fraps YUV source video; output viewed with...
          • YouTube and most media players displayed correct colors (= ok)
          • VDB and PE4 = blown-out (y+)
          • SM:G and FF = green push (g+)
            I didn't look into workarounds; maybe later
      • codec: mpeg4 (MPEG-4 ASP)
        • always add "-pix_fmt yuv420p"
        • Fraps YUV source video; output viewed with...
        • Fraps RGB source video; output viewed with...
          • SM:G, FF = ok
          • all others, add "-vf colormatrix=bt601:bt709"
      • codec: libx264
        • always add "-pix_fmt yuv420p"
        • Fraps YUV source video; output viewed with...
          • VDB, PE4, SM:G, FF, add "-vf colormatrix=bt709:bt601"
            BTW, "-colorspace 6" would have been slightly better, since
            "-vf colormatrix" changes the video data, but "-colorspace" only
            adds a tag; but unfortunately, "-colorspace" seemed to have no effect.
          • all others = ok
        • Fraps RGB source video; output viewed with...
          • VDB, PE4, SM:G, FF = barely perceptible red push
          • all others = r+
            I didn't look into workarounds; maybe later; I think would
            write an Avisynth script to bring Fraps RGB into ffmpeg
      • codec: libx264rgb
        • Fraps RGB source video; output viewed with...
          • PE4 = r+
          • MP:H, YT = error (scrambled video)
          • all others = ok
      • codec: huffyuv (lossless)
        • Fraps YUV source video; output viewed with...
          • VDB, PE4 = ok
          • MP:H, MP:R, MP:W, YT, add "-vf colormatrix=bt709:bt601"
        • Fraps RGB source video; output viewed with...
          • VDB[1], PE4, MP:H, MP:R, MP:W = ok (add "-pix_fmt rgb24")
          • others not tested
    continued in next post...
  4. raffriff

    raffriff Moderator Staff Member Site Contributor

    • "VDB" (VirtualDub 1.10.2)
      • add the Alias Format filter (set color space = Rec.709; range = no change).
      • codec: ffdshow/mjpeg
        • (required a complicated setup for mediocre results)
      • codec: DiVX; XViD; Microsoft MPEG-4 ASP; MS VC-1
        • Fraps YUV source video; output viewed with...
          • MP:H, BS:O = r+
          • others not tested
            didn't spend much time with these codecs
      • codec: x264vfw
        • Fraps YUV or RGB source video; output viewed with...
      • codec: Huffyuv (lossless)
        • Fraps YUV source video (Huffyuv mode= YUY2); output viewed with...
          • VDB = ok
          • PE4 = freezes[2]
          • MP:H, MP:R, MP:W, YT = r+
            will explain workaround later maybe
        • with Fraps RGB source video, (Huffyuv mode= RGB)
          • VDB, MP:H, MP:R, MP:W = ok
          • others not tested
      • codec: Lagarith (lossless)
        • with Fraps YUV source video (Lagarith mode=YV12); output viewed with...
          • VDB, PE4, MP:H, MP:R, MP:W = ok
          • YT = error (upload failed)
          • others not tested
        • with Fraps RGB source video (Lagarith mode=RGB); output viewed with...
          • VDB, PE4, MP:H, MP:R, MP:W, BS:O, FF = ok
          • SM:G = error (crash, IIRC)
    VDB[1]=must open with DirectShow plugin
    freezes[2]=need to test again with PCM audio instead of MP3

    continued in next post...
  5. raffriff

    raffriff Moderator Staff Member Site Contributor

    • Avidemux 2.5
      • codec: x264.exe
        • Fraps YUV source video; output viewed with...
          • VDB = y+ and g+
          • MP:H = y+
            could not find a workaround
    • Handbrake
      • codec: x264.exe
        • Fraps YUV source video; output viewed with...
          • VDB[1], MP:H = ok
          • others not tested
    • "MS-Expr" (Microsoft Expression Encoder)
      • codec: Microsoft WMV9 VC-1 Advanced
        • Fraps YUV source video; output viewed with...
          • VDB[1], PE4, SM:G, FF = ok
          • all others = r+
    • "PE4" (Premiere Elements 4)
      • codec: x264vfw
        • Fraps YUV source video; output viewed with...
          • VDB = ok
          • MP:H, add "--colormatrix bt709"
          • others not tested
          • very high bitrate for some reason
    • "PE4->VDB" (Premiere Elements 4, frameserving to VirtualDub)
      • codec: x264vfw
        • Fraps YUV source video; output viewed with...
          • VDB = ok
          • MP:H, add "--colormatrix bt709"
          • others not tested
    • StaxRip
      • input script for Fraps sources: ConvertToYV12(matrix="Rec709") [see here]
      • codec: x264.exe
        • x264: set colormatrix=bt709 (did not seem to have any effect)
        • Fraps YUV or RGB source video; output viewed with...
          • VDB, MP:H, YT, BS:O = ok
          • MP:W = y-
          • SM:G = g+
    NOTE: Any time MP:H is listed but YT is not, that's because YT always (with one exception) followed MP:H. The exception was a test video uploaded at standard def (848x480), which was ok on MP:H but r+ on YT.

    continued in next post...
  6. raffriff

    raffriff Moderator Staff Member Site Contributor

    An interesting post on doom9.org by Petrakeas2, "BT.601 and BT.709 compatibility benchmark," confirms most of my tests, and my tests confirm most of his.

    I will try to sum it all up here, if possible...

    There are six ways to handle color space, given YUV input. RGB input (such as produced by the Fraps decoder) is not affected.

    1: Always use Rec.601 (the old standard-definition color space); if given Rec.709, display it with green push.​

    2: Always use Rec.709 (the high-definition color space); if given Rec.601, display it with red push.​

    3: Use Rec.709 for HD only: if resolution is higher than a certain threshold (approximately 480p), assume Rec.709 (display Rec.601 as red); else assume Rec.601 (display Rec.709 as green).​

    1a, 2a, 3a: Whether using strategy 1, 2 or 3, use color-space tags in the video stream, if available, to decide. If the tags are correct, great. If not, the colors will be wrong.​

    I didn't test methodically to see which strategy each player was using; for one thing, I was only interested in high definition - 720p or above, although I did run one quick test at 480p. Therefore I can't tell (and am not too interested in) the difference between strategy 3 (Rec.709 for HD only) and strategy 2 (always use Rec.709). Petrakeas2 goes into SD behavior in more detail.

    Here's how I see the behavior of the applications I tested:
    • VirtualDub always assumes Rec.601, unless the Rec.709 tag is added (x264 "--colormatrix bt709"). Otherwise, Rec.709 input (g+) can be corrected with the Alias Format filter (set color space = Rec.709; range = no change). To see the correction in still-frame, at least one other color filter must be active. I suggest the HSV filter, left at default settings.
    • Premiere Elements 4 always uses Rec.601. Not aware of any way to correct it. Not sure how newer versions work, but see the note about Adobe CS5.5 below.
    • Media players using the Haali renderer seems to use Rec.709 for HD only. (I find the choice of video renderer matters more than the media player application itself)
    • Media players using the VMR9 renderer, whether the "window" flavor or the "renderless" flavor, seem to use Rec.709 for HD only.
    • Media players using the Overlay option seem to use Rec.709 for HD only, but they display reduced luma range (washout) with WMV9/V-1 in my tests; also with StaxRip/x264 output, but only with Fraps RGB sources.
    • YouTube seems to assume Rec.709 always...
      • Always Rec.709 in HD for me; Rec.709 in SD based on only one test (with x264)
      • Note this can vary depending your hardware acceleration settings, and possibly other factors. My findings here disagree with Petrakeas2's, just so you know. I am using WinXP and a NVidia DX9 GPU; Flash player has hardware acceleration enabled.
      • More testing, on a variety of systems, is needed. You can look at my test videos on YouTube and tell me if you see what I am seeing.
      • I agree with Petrakeas2 that having multiple Flash windows open will affect the color; only the first window will have hardware acceleration.
      • If my tests are valid, using a media player with the Haali renderer is the best way to "preview" the color as it will appear on YouTube, assuming 720p resolution or more.
    According to Petrakeas2 (and I believe him),
    • Adobe Suite CS5 always use Rec.601, but CS5.5 uses Rec.709 for HD only.
    • Sony Vegas uses Rec.709 for HD only. I believe there are filters to correct this if needed -- anyone know?
  7. raffriff

    raffriff Moderator Staff Member Site Contributor

    (place holder)
  8. raffriff

    raffriff Moderator Staff Member Site Contributor

    Bump - finished!
    Feedback is welcome. Your questions will help me make this more clear, I hope - and more correct, if I'm wrong somewhere. :)

    Again, if you aren't sure how this works, or if your encoding application is not listed, post a reply and I will help you with it.

    - - - - - - - - - - - - - - - - - - - - - - -
    EDIT Aug 8, 2012 - nothing huh? I guess you see a wall of text and step away. I don't blame you.
    I blame myself for my unreadable style. I'll try again:
  9. Now i am not absolutely sure, but doesn´t Fraps work quite, obscure?
    It does like this:

    Gameplay(RGB) - > Fraps -> RGBtoYV12 rec.601 - > Compress(Fraps codec).
    If you want to play the file, or encode etc

    Video(YV12 rec.601) - > Fraps Decoder YV12toRGB -> View video or Encode Video - > YV12 (choose Rec.601/Rec.709).

    It´s a very bad way overall, the file is YV12, but the only way to use it, is with RGB, and then you have to convert it again to YUV, and you will have to choose your Rec, and as your PC is sRGB it goes to Rec.709, but i think the original was Rec.601, meaning a colormatrix conversion is done through the whole loop, along with additional chroma loss from Upscaling to RGB.

    I may be wrong, but i think this is how it works with Fraps.
  10. raffriff

    raffriff Moderator Staff Member Site Contributor

    The Fraps codec always decodes as RGB, whether the recording was made in RGB or YUV.

    Fraps YUV format is full range, Rec.709. Source: Fraps developer via doom9.

    I had to write my own DirectX test pattern generators to confirm this. Here is my latest version: Frafs Test Pattern 0.5

    FFmpegSource2 (requires Avisynth) decodes Fraps as YUV or RGB as required; this is a better way to go, as the Fraps color resampling (YUV to RGB) is a bit soft. With access to the original YUV, you can apply your own chroma resampling method when you convert to RGB.

    If you see colors wrong when decoding as full range 709, it's for one of three reasons, that I know about.
    1. You have not calibrated your playback signal chain. I address that and more in Fraps colorspace issues (wrong color, faded, dark etc) - part 2, and before that, in the Fraps 'dark' thread, post #40 and post #52.
    2. You have color management software running which needs to be disabled during capture.
    3. The game is doing gamma manipulations post frame buffer. You can try adjusting "brightness" in the game, and/or you must color correct in post.
  11. Yes, it always decode and upsample to RGB, which causes converting to YUV again to be a quality loss.
    Of course, if you use Editing software that work in RGB, that will cause similar effect, but if you work in Avisynth, it´s just a loss.

    I know that ffmpegsourc2 can decode Fraps directly to YV12, and i guess it shows the true image then, meaning, no upsampling/downsampling etc, though i have not done any deep testing.

    I am not getting wrong colors, you will get the correct colors if you decode as rec.709 full range.
    But, the problem is, when fraps encode the image that is shown, for example a game.

    It will take that image, convert it to YV12 With Rec.601 i think. And there is a loss of color.
    When you decode it using Rec.709, it will look right, but i think you will get rounding errors in the colormatrix cause it was save in the wrong colormatrix to begin with.

    you can´t make rec.709 to rec.601 and vice versa, losslessly.

    That is the problem, though i don´t think anyone really cares, as the rounding errors aren´t a big deal.
    But if it is done this way, then the errors are there.
  12. raffriff

    raffriff Moderator Staff Member Site Contributor

    >It will take that image, convert it to YV12 With Rec.601
    Why would you do that :/
    FFmpegSource2("video.avi", atrack=1)
    ## OPTIONAL: convert to RGB only if required by downstream filters etc
    #ConvertToRGB32(matrix="PC.709", chromaresample="spline16")
    This assumes the signal destination handles full range 709 correctly. For example, VirtualDub expects Rec.601, so you must use the Alias Format filter; Color=Rec.709, Range=No Change.

    >you can´t make rec.709 to rec.601 and vice versa, losslessly
    That is absolutely correct. So try to convert as seldom as possible; don't go back and forth. Same for luma range and for YUV<>RGB. Eventually, it will probably have to be YUV, studio range in order to be encoded and viewed (ignoring 4:4:4 for now), but when this conversion is done depends mainly on your video editor. What does it expect to see? Trying to answer that question was the genesis of this thread.

    > i don´t think anyone really cares
    I've kind of made it my business to care, but hopefully, not to an obsessive degree :)
  13. raffriff

    raffriff Moderator Staff Member Site Contributor

    I have :D Some of my test images - compare:

    test image 20.png test image 21.png test image 22.png

    (Fraps YUV ... FFMpegSource ... Fraps RGB)
    FFMpegSource is very slightly sharper; RGB is a whole lot sharper.
  14. Hi, i know this is an old Thread, but i have a simple Question.

    I record with Fraps in YUV, how should i encode now with Avisynth+X264 for the correct colors?

    video1 = avisource(x).ConvertToYV12(matrix="Rec709")
    video2 = AudioDub(FFVideoSource(X), FFAudioSource(X)).ColorYUV(levels="PC->TV")

    Both seems to be the same in VDub and when i open the source with Media Player Classic to check, it's still looking good.
  15. raffriff

    raffriff Moderator Staff Member Site Contributor

    Hi Bregor, either way is OK, but I would add the x264 command --colormatrix bt709, or equivalent GUI option if using StaxRip etc; this is not always needed (depending on the media player you are using, since many assume Rec709/BT709 by default), but it never hurts.

    Instead of .ColorYUV(levels="PC->TV"), try the VirtualDub Alias Format Filter, range=full instead; the quality is a little better.

    All in all, I prefer FFVideoSource/FFAudioSource, as the colors are a wee bit sharper; plus ConvertToYV12 is not needed.

    Regarding VirtualDub: VirtualDub always previews YUV assuming Rec609 - so greens look a more intense, and reds a little less, than they will on the final video. To fix this, add an Alias Format Filter, color space=Rec.709 - for previewing purposes ONLY. Disable before encoding. This does not apply to previewing RGB video.
  16. Thanks for your answer.

    So Fraps YUV is "rec709, Full Range", as you stated in a post by a quote from a developer of Fraps. I don't understand how that could fit together, because rec709 should be TV-Range, or not?

    Assuming Fraps YUV is PC.709 (Full Range with 709 matrix) and i encode it with rec.709 (tv range with 709 matrix), shouldn't there be a differenz between the source and the encoded video in the same player?

    This is what still confuses me.
  17. raffriff

    raffriff Moderator Staff Member Site Contributor

    It's a little very confusing. I went and consulted the source code, so I'm pretty sure this is correct:

    matrix="Rec709" assumes RGB is 0-255 and YUV is 16-235;
    so full-range RGB->ConvertToYV12(matrix="Rec709")
    gives TV-range YV12,
    and TV-range YV12->ConvertToRGB32(matrix="Rec709")
    gives full-range RGB.
    This is usually what you want. In the case of the Fraps codec,
    which is full-range RGB, the correct conversion to TV-range YV12 is performed.
    Rec601 works the same way, but the colors are slightly different.

    matrix="PC.709" keeps the same luma range, which is not used very often.
    It might be useful for converting Fraps+FFVideoSource full-range YV12 to RGB
    without exaggerating the luma range.

    BTW for YUV<->YUV and RGB<->RGB conversions, matrix has no effect.

    EDIT: tested and verified.
    Last edited: Nov 5, 2014
  18. OK, so when i encode with ConvertToYV12(matrix="Rec709") i'll get e YV12 TV-Range Video. This is what we want for flash player and others, like you said. check?

    So the chain in Avisynth is like:
    Avisource("bla.avi") --> ConvertToYV12(matrix="Rec709")
    Fraps YV12 (TV-Range, 709) --> RGB(Full-Range, 709) --> YV12(TV-Range)

    and for FFMS it should be
    Fraps YV12(TV-Range, 709) --> YV12(TV-Range,709)

    The Problem i have is this: with the second processchain my video gets much darker (wrong luma?) than the source on WMP or MPC. But with ColorYUV(levels="PC->TV") it looks alright, like the source. This means that Fraps YV12 is not YV12(TV-Range), like assumed but Full-Range instead?

    Maybe i misread some of your posts, but i am still lost here. For me, avisource and rec709 works, but i want to understand. Sorry. :)
  19. raffriff

    raffriff Moderator Staff Member Site Contributor

    No problem, there are probably other people with the same questions. You are very close:
    Last edited: Nov 5, 2014
    Thalmor Wizard likes this.
  20. Aha, so i have to convert to rec709 because most players assume that i have a rec709 video, but without the conversion PC-->TV they will fail in decoding everything corretly.
    raffriff likes this.

Share This Page