re-encode on the fly

Discussion in 'FRAPS General Discussion, Guides, and Tutorials' started by billman, Mar 26, 2013.

  1. billman

    billman Site Contributor Well-Known Member

    This is pretty rough to do to your PC but I have written a short script in Autohotkey to allow for almost endless recording with fraps. I can manage a solid 30 FPS, occasional drops but that's expected. Note my fraps location is 3 x 500 GB WD blacks in windows stripe raid and I'm using and I7 3700 CPU and both get there ass worked.

    Code:
    delsource = true
    #include Affinity.ahk
    top:
     
    Sleep 1000
    Loop, F:\fraps\*.avi
    {
    source = %A_LoopFileFullPath%
    destination = C:\out4\%A_LoopFileName%.mp4
    Sleep 1000
    run, "C:\Program Files (x86)\Handbrake\HandBrakeCLI.exe" -i "%source%" -o "%destination%" --encoder x264 -x264opts keyint=1 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0 , ,Hide , PID_HANDBREAK
    Affinity_Set( 3, PID_HANDBREAK )
    Sleep 300
    Process, WaitClose, %PID_HANDBREAK%
    if delsource = true
    IfExist , %destination%
    FileDelete %A_LoopFileFullPath%
    }
    goto top
    this is the first functional version have made,
    It polls my fraps location for new files, then send it to Handbreak CLI
    after the encode it deletes the source file to ensure your fraps location does not fill up.
    this creates aprox 150mb files from 4GB chunks however the color gets a bit messed but is corrected by the video editing suite. I need to do some more testing of different encode settings in handbreak but though this was worth sharing.

    My purpose for this was I have been challenged to make a 100 hour YouTube video of my game play.
    I'm a bit over half way there and still have more than sufficient space to continue collection on a single 500GB drive.

    I'm keen to get some feedback about this. I have found a method of limiting handbreakCLI to a single thread permanently using ImageCFG.exe allowing my game to run smoothly during this process.

    If there is any interest I will add a GUI for setting locations and encode quality.

    My previous works if you care http://www.youtube.com/user/billman1987
     
  2. raffriff

    raffriff Moderator Staff Member Site Contributor

    Very interesting. I'd like to try it, but I know my system can't run a game, Fraps and Handbrake all at the same time.

    What about the "file locking" issue, though? Doesn't Fraps prevent you from deleting any file while recording is going on? (While Fraps is recording, all files in the "session" (or group of 4GB files) are "locked" by the Fraps process. At least they were, last time I checked)

    Other threads by people doing copying or re-encoding on the fly, which might interest you:
     
  3. billman

    billman Site Contributor Well-Known Member

    Thanks for your interest, I am using fraps version 3.5.9 and It indeed locks the files until it is finished, this is a good thing as my script will fail silently until the file is accessible, even if you select break into 4 GB chunks it will not allow you to remove the file until you stop capture, I am thinking about going back to an earlier version that is only capable of creating the 4GB chunks as it would finalise each video fly rather than wait until you stop recoding.

    The RAM drive option is not viable for me as I tend to create up to 60 GB of video before Fraps realise the files.

    Auto Copy is also not a good Idea as it will push the Fraps drive too hard If your trying to do full speed Write then a Full speed Read at the same time, even my RAID dose not handle this resulting in massive frame drop. With Handbreak or other command line encoding utilities cause heavy CPU usage (I limit to one core) but limit HDD usage. Resulting in about a 50/50 read/write scenario.

    I know why I get frame drops, my windows raid only has so much cache and requires CPU (that is at 100 percent) to do work, If I used a real raid card this problem would go.

    I see many other have this Idea seems I am one of the few pulling it off. My next problem is rendering over 24 hours in Sony Vegas seems to cause failure. It would seem I still have a few technical hurdles to get over but I will get there, I will have to browse this forum some more there is lots of different solutions here.
     
    raffriff likes this.
  4. billman

    billman Site Contributor Well-Known Member

    An update if anyone is following, I discovered HandbreakCLI while limited to one CPU is not quite fast enough to transcode at 30 FPS and at moments is causing almost a hard lock of the system for 5-30 seconds on seeming random occasions. Sony Vegas was causing the same thing when limited to one CPU. So at the moment my read/write rate is about 30/70 this has extended my max record time from 6 hours to almost 12. This would normally be enough but I'm not normal.

    Next step is finding away to speed up Handbrake without degrading the footage, I am going to try and only allow Handbrake 2 CPU's, if that fail I guess I will have to use faster encode presets but at a cost of large output, almost defeating the purpose of what I'm doing.

    Maybe I'm just dreaming, Recording a game, Transcoding Fraps Footage to MP4 and Rendering in Sony Vegas at once might just be a bit hard to do all at once, by god I will kill this computer trying.

    Other interesting thing I found with Sony Vegas 11 is that the render seems to fail at about at about 14 hours thou the video or is failing once the output file hits about 64GB in size. So as it stands I am rendering 10 hour chunks each about 50 GB. That seems extreme but the output is beautiful considering 10 hours of fraps exceeds my 1.5 TB storage but I think I am going to need an additional HDD just to store the rendered output my OS drive is getting full fast.
     
  5. raffriff

    raffriff Moderator Staff Member Site Contributor

    Rendering files that large in Vegas, I think you would run into a RAM shortage problem.

    With Handbrake or any x264 command line, use the "ultrafast" preset for maximum speed.

    For screaming fast encodes with multicore processors, add "--keyint 1" to the command line (you might have to use the syntax variant "-x264opts keyint=1" with Handbrake). This makes each video frame stand on its own instead of being part of a large group (GOP). It allows each CPU core to work on a separate video frame, without waiting for other frames to finish processing. So depending on the number of cores, you may see another doubling of speed - or more.
    https://encodingtalk.com/threads/fraps-to-youtube-with-h-264.894/#post-4756
    It also vastly cuts RAM requirements. There is a file size penalty: you will have from 25-50% larger files.
     
    billman likes this.
  6. billman

    billman Site Contributor Well-Known Member

    Thank you for them suggestion on syntax for handbrake, I try that for my next round or recoding.
    Sony Vegas top out at 2GB and Handbrake tops out 600MB, SAMP tops about 400MB of ram however I do seem to max about 6GB of "modified" RAM usage (files waiting to be written) and "Standby" eats the rest while this all this is going on at once. So I am getting close to using my 16GB.

    I Set fraps back to Single File output and some of the Handbrake Transcodes are failing. Most notably if the resulting video file exceeds 4GB as the largest successful encode is 3.58 GB and 43 minutes long. So breaking Fraps output into 4GB chunks is my best bet for error free encoding or maybe another syntax option for handbrake would solve this.

    EDIT: I found where I went wrong. If only I could read.

    From the CLI guide itself

    "-4 or --large-file if you do not enable it, and your MP4 is larger than 4 GB, the output will be unplayable and unrecoverable."


    Some File Stats

    Fraps Recordings - 222 GB
    Resulted in 5.7GB failed encode
    [​IMG]



    Successful Handbrake Output, - 3.58 GB
    [​IMG]


    10 hour Sony Vegas output. - 50.6 GB
    [​IMG]
     
  7. raffriff

    raffriff Moderator Staff Member Site Contributor

    Okay, great. I'm surprised Vegas does so well. One thing, it's better to let Fraps make 4 GB segments IMO because if the computer crashes during recording you could lose your entire video versus losing only the final segment.
     
    Thalmor Wizard likes this.
  8. billman

    billman Site Contributor Well-Known Member

    Processing FRAPS output directly was troublesome but these MP4's I'm using have been hassle free in Vegas (over 20 hours of footage processed without even reopening it) I used to save the project and reopen just to render previously, 4GB chunks are probably better however this computer has never crashed during recording despite the ass whooping I give it and having large continues files are easier to edit (sure you know that).

    I think I solved the issues with the oversize MP4 with one switch, However now I have a drama with CPU affinity and handbrakeCLI.

    I used IMAGECFG.exe to set it to Unicore to avoid it alone reducing the performance of the game while recording and undoing it should be a matter of replacing the modified EXE, but removing and reinstalling handbrake has not undone the affinity changes. I tested the same thing on handbrake.exe and replacing the EXE and doing so removed the affinity restrictions, but this is not true for CLI. WTF! 32bit complex or maybe cause I call it from Autohotkey this has me stumped.

    I guess I will need to use "start /AFFINITY 3 handbrakeCLI.exe" (limit to CPU 0 and 1) however "WINWAIT" only waits until "START" has finished before running the next iteration of the AHK loop. This causes carnage (over 15 encodes at once w00t) so I can only foresee I have to make a more complex script to detect the presence of CLI and to pause the script before starting the next encode. Not hard I'm just lazy.

    Also the the suggestion of single frame encoding plus a few other tweaks, took average encode FPS from 15 to 19 on single core, 25 to 40 on 2 Cores and now upto 50 FPS if allowed to have the whole CPU, File size chucks went from 150MB to 300MB for a 1 min 45 second video which is acceptable for me. This has also improved the quality of the output which I did not expect. I though faster encoding would produce substandard footage. I'm getting close to my goal.
     
  9. raffriff

    raffriff Moderator Staff Member Site Contributor

    billman likes this.
  10. billman

    billman Site Contributor Well-Known Member

    That did the trick perfectly. Used Process, WaitClose, to pause the script after setting affinity with that method.
    After 3 hours of recording I have produced 18.5 GB of transcoded footage, only 10 minutes of footage remained in the fraps folder after recording stopped. A few frame drops to tidy up and I'll be happy.

    Thanks again.

    Output Frames Looking GOOD
     
    raffriff likes this.
  11. billman

    billman Site Contributor Well-Known Member

    I rebuilt my bastard.ahk as the old one got stupid. This one has a gui for setting -
    Source location
    Output Location
    Fraps.exe location
    HandbrakeCLI.exe location

    and has a go button, closing the GUI also stops the associated HandbrakeCLI.exe, checks to make sure the output file is a reasonable size, but I think my math is wrong here.

    Code:
    #SingleInstance force
    Action = Startup
    Gui, +Resize
    Gui, Margin , 0, 0
    Menu, FileMenu, Add, &Go,              Menu_Go
    Menu, FileMenu, Add, &Source,          Menu_Source
    Menu, FileMenu, Add, &Destination,  Menu_Destination
    Menu, FileMenu, Add, &Fraps,          Menu_Fraps
    Menu, FileMenu, Add, &Handbrake,      Menu_Handbrake
    Menu, MyMenuBar, Add, &File,        :FileMenu
     
    Menu, SettingsMenu, Add, &Fraps,          Settings_Fraps
    Menu, SettingsMenu, Add, &Handbrake,      Settings_Handbrake
     
     
    Menu, MyMenuBar, Add, &Settings,    :SettingsMenu
    Gui, Menu, MyMenuBar
    Gui, Add, ListView, xm h400 w700 vMyListView gMyListView , Time|Action|Output
    Gui, Show , AutoSize gListView vListView, bastard
    LV_Add("", A_Now, Action,A_ScriptFullPath)
    gosub postmessage
     
    gosub startup
    return
     
    Settings_Handbrake:
    MsgBox comming soon
    return
    Settings_Fraps:
    run "%fraps%"
    return
     
     
    Menu_Go:
    IfEqual source
        gosub Menu_Source
    IfEqual destination
        gosub Menu_Destination
    IfEqual fraps
        gosub Menu_Fraps
    IfEqual handbrake_CLI
        gosub Menu_Handbrake
    gosub check_source
    IfEqual handbrake_syntax
    handbrake_syntax = --encoder x264 -x264opts keyint=1 -4 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0
    return
     
    Menu_Source:
    FileSelectFolder, source , , 0 , Select Source Location
    if errorlevel 1
    return
    Menu_Destination:
    FileSelectFolder, destination , , 1 , Select Destination Location
    return
    Menu_Fraps:
    ;FileSelectFolder, fraps_dir , , 0 , Select Fraps.exe Location
    fraps = C:\fraps\fraps.exe
    IfNotExist fraps.exe
        goto Menu_Fraps
    return
    Menu_Handbrake:
    FileSelectFolder, handbrake_dir , , 0 , Select HandbrakeCLI Location
    handbrake_CLI = %handbrake_dir%\HandBrakeCLI.exe
    IfNotExist %handbrake_dir%\HandBrakeCLI.exe
        goto Menu_Handbrake
    return
     
    transcode:
     
    loop, parse, source_tobe_processed , `n ,
    {
        Action = Transcode
        IfNotEqual, A_LoopField,
        {
        LV_Add("", A_Now, Action,A_LoopField)
        gosub postmessage
       
        Run %handbrake_CLI%  -i "%source%\%A_LoopField%" -o "%destination%\%A_LoopField%" %handbrake_syntax%,, Hide , handbrake_pid
       
        Process, wait, HandBrakeCLI.exe
        Results = HandbrakeCLI PID-%handbrake_PID%
        LV_Add("", A_Now, Action,Results)
        gosub postmessage
       
        Affinity_Set( 3, handbrake_pid )
        Process, WaitClose, HandBrakeCLI.exe
        Action = Complete   
        LV_Add("", A_Now, Action,A_LoopField)
        gosub postmessage
        gosub check_output
        gosub delete_source
        }
    }
    Action =
    Sleep 10000
    goto check_source
    return
     
    delete_source:
    Action = Delete Source
    if check_output_lock = locked
        return
    IfExist , %destination%\%A_LoopField%
    {
        FileGetSize, output_size , %destination%\%A_LoopField%, M
        FileGetSize, source_size , %source%\%A_LoopField%, M
        output_ratio = %source_size%
        output_ratio /= %output_size%
        Results = Input Video %source_size%MB Output Video %output_size%MB Reduced to %output_ratio% Percent.
        LV_Add("", A_Now, Action,Results)
        gosub postmessage
        if delete_source = true
            if output_ratio > 8
            {
                FileDelete , %source%\%A_LoopField%
                Results = %A_LoopField% Successful
                LV_Add("", A_Now, Action,Results)
                gosub postmessage
            }
            else
            {
                Results = %A_LoopField% Too Small Did Not Delete
                LV_Add("", A_Now, Action,Results)
                gosub postmessage
            }
    }
    Action =
    return
     
     
    source_tobe_processed:
    Action = Processing
     
    loop, parse, source_tobe_processed , `n ,
    {
        IfNotEqual , source_tobe_processed
        {
            LV_Add("", A_Now, Action,A_LoopField)
            gosub postmessage
        }
    }
    Action =
    goto transcode
    return
     
    check_output:
    Action = Check Output
     
    FileReadLine, check_output, %destination%\%A_LoopField%, 1
        if errorlevel = 0
        {
            check_output_lock = Unlocked
            LV_Add("", A_Now, Action,check_output_lock)
            gosub postmessage
        }
        else
        {
            check_output_lock = locked
            LV_Add("", A_Now, Action,check_output_lock)
            gosub postmessage
        }
    return
     
     
    check_source:
    Action = Check Source
    Loop, %source%\*.avi
    {
        LV_Add("", A_Now, Action,A_LoopFileName)
        gosub postmessage
        FileReadLine, check_source, %A_LoopFileFullPath%, 1
        if errorlevel = 0
        {
            check_source_lock = Unlocked
            source_tobe_processed = %source_tobe_processed%`n%A_LoopFileName%
        }
        else
            check_source_lock = Locked
        LV_Add("", A_Now, check_source_lock,A_LoopFileName)
        gosub postmessage
    }
    Action =
     
    goto source_tobe_processed
    return
     
    startup:
    Action = Output Destination
    delete_source = true
    RegRead destination , HKCU, Software\Fraps3 , destination
    LV_Add("", A_Now, Action,destination)
    gosub postmessage
    Action = HandbrakeCLI
    RegRead handbrake_CLI , HKCU, Software\Fraps3 , handbrake_CLI
    LV_Add("", A_Now, Action,handbrake_CLI)
    gosub postmessage
    RegRead handbrake_syntax , HKCU, Software\Fraps3 , handbrake_CLI_syntax
    Action = Handbrake Syntax
    handbrake_syntax = --encoder x264 -x264opts keyint=1 -4 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0
    LV_Add("", A_Now, Action,handbrake_syntax)
    gosub postmessage
    Action = Fraps
    RegRead fraps , HKCU, Software\Fraps3 , fraps
    LV_Add("", A_Now, Action,fraps)
    gosub postmessage
    Action = Source Location
    RegRead source, HKCU, Software\Fraps3 , source
    LV_Add("", A_Now, Action,source)
    gosub postmessage
    Action =
     
    return
     
    F1::
    RegWrite:
    RegWrite, REG_SZ, HKCU, Software\Fraps3 , source, %source%
    RegWrite, REG_SZ, HKCU, Software\Fraps3 , destination, %destination%
    RegWrite, REG_SZ, HKCU, Software\Fraps3 , handbrake_CLI, %handbrake_CLI%
    RegWrite, REG_SZ, HKCU, Software\Fraps3 , handbrake_syntax, %handbrake_syntax%
    RegWrite, REG_SZ, HKCU, Software\Fraps3 , fraps, %fraps%
     
    return
     
     
     
    PostMessage:
    PostMessage, 0x115 , 1, 1, SysListView321, bastard, ,,
    LV_ModifyCol()
    return
     
    GuiSize:
        if A_EventInfo = 1
        {
        return
        }
        GuiControl, Move, MyListView, % "W" . (A_GuiWidth) . " H" . (A_GuiHeight )
    return
     
    MyListView:
    if A_GuiEvent = DoubleClick
    {
    return
    }
    return
     
     
     
    GuiClose:
    gosub RegWrite
    Process, Close, %handbrake_PID%
    ExitApp
     
     
     
     
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    Affinity_Set( CPU=1, PID=0x0 ) { ; CPU0=1 CPU1=2 | to use both, CPU should be 3
      Process, Exist, %PID%
      IfEqual,ErrorLevel,0,  SetEnv,PID,% DllCall( "GetCurrentProcessId" )
      hPr := DllCall( "OpenProcess",Int,1536,Int,0,Int,PID ) 
      DllCall( "GetProcessAffinityMask", Int,hPr, IntP,PAM, IntP,SAM )
      If ( CPU>0 && CPU<=SAM )
        Res := DllCall( "SetProcessAffinityMask", Int,hPr, Int,CPU )
      DllCall( "CloseHandle", Int,hPr )
    Return ( Res="" ) ? 0 : Res
    }
    
    Finally starting to do some error handling but needs much more work yet.
    But here is a screen for anyone following my progress.
    [​IMG]
     
  12. raffriff

    raffriff Moderator Staff Member Site Contributor

    Looks nice. I like a program with a clear & readable log file.

    you mean "if output_ratio > 8" ? If you are checking for a valid conversion, maybe you need to run MediaInfo CLI on both files and compare durations...
     
  13. billman

    billman Site Contributor Well-Known Member

    MediaInfo CLI sound like it should be the solution I was initially after. I was wondering how I could get the duration without writing my own binary reader or doing ridicules DLL calls, which is not all that hard to do just marjory time consuming. Looks like I might need an installer, or feature to grab Handbrake and MediaInfo from a remote source if not present. It was very early in the morning when I wrote the compare function and watching the output last night in the log proved it was bogus but still provide a baseline for me to work with. The log has been the biggest time consuming part of this script to make. But I am learning the fundamentals of state tracking and error handling, so it a bit buggy and wild but provides clear feedback on exactly what is going . Some more thought into the layout is needed but functionality take precedence over prettiness in my world.

    I hope this makes sense to anyone but me, my brain is a bit fried I just finished a 15 hours gaming/recording stint and recorded about 90 percent of it, proving itself a very handy bit of script. However I have tracked my frame drop issue to the finalisation of the HandbrakeCLI encode. It would seem to me that having handbrakeCLI limited to 2 of 4 cores causes the CPU to be unable to do anything else during this time. I don't see any lockups when handbrake is give the whole CPU but I do struggle to record at 30FPS with this setting . On occasions it so bad the entire PC appears lockup for up to 5 seconds and shutter for up to 15 seconds. I'm guessing it is the L3 Cache with its token ring feature that enables peripherals to talk directly to the CPU cache. This cache is getting held up by the affinity setting. As each core requires access to the L3 cache disallowing the process to be pushed to the next core in the token ring it is physically halting the other cores until the current process is completed. This is only speculation of course but I can not find a spike in CPU time, HDD bandwidth, or excessive ram usage. Infact CPU usage appears to drop to zero during this time. Also and breaking the fraps output into 4gb chunks caused this to happen more frequently with slightly less severity. A FRAPS AVI of 300GB does not seem to prolong the shuttering significantly more than a 4GB FRAPS AVI .If I stop the recording during a shutter it take about 2 seconds from the time i hit the hotkey to the time my system comes back or I can allow it to record during the shutter and the audio plays and records fine (Game sound loops but streamed music is consistent). So far this has not caused any crashing or loss of connection to servers so that tells me the PCH (Platform Controller Hub) and peripherals is still ticking at normal rate during this event.
    So an older Core 2 Quad may not have this issue. But it is the new cache system that provides cool features like the in chip video processor and intel quick sync and other hardware acceleration techniques that provide the biggest performance improvement over it predecessors .

    These images should help you visualise how the Shared L3 Cache could cause it own bottleneck.

    [​IMG]
    Above is a Quad-core Kentsfield package (2006)

    [​IMG]
    Above is a Quad-core Sandy Bridge die (2011)

    In my opinion forcing a quad core chip to talk to a single cache is a step backwards in terms of true parallelism but it benchmarks that sell chips not real world performance. Funny enough the old core 2 quads still stand up to todays games assuming you got some sweet ram and current video card. Enough of that.

    I really like handbrake (took me ages to figure it out) but I now need to experiment with different command line based encoding solutions.

    and raffriff thanks for taking a look at my script it is rather messy and I have already made some major changes to the order and way each feature is called, I don't feel quite so alone on my journey of technical discovery knowing that someone actually understand what I'm doing and why.
     
    raffriff likes this.
  14. raffriff

    raffriff Moderator Staff Member Site Contributor

    Have never thought of that, but it seems logical. Brilliant!
     
  15. billman

    billman Site Contributor Well-Known Member

    Ok I have got a video to share, created but this process of encoding on the fly.
    Frame quality is spectacular even once being uploaded to youtube.



    Sony Vegas does not like the MP4's made by hand break making the editing a little harder than normal.
     
  16. I would like to fraps on a ramdisk then recode to a hdd and delete the 4GB source file once recoded with handbrake.

    Is it possible with this ahk ?
     
  17. ETAdmin

    ETAdmin Administrator Staff Member

    I believe so - check post #11
     

Share This Page