re-encode on the fly

billman

Well-Known Member
Site Contributor
This is pretty rough to do to your PC but I have written a short script in Autohotkey to allow for almost endless recording with fraps. I can manage a solid 30 FPS, occasional drops but that's expected. Note my fraps location is 3 x 500 GB WD blacks in windows stripe raid and I'm using and I7 3700 CPU and both get there ass worked.

Code:
delsource = true
#include Affinity.ahk
top:
 
Sleep 1000
Loop, F:\fraps\*.avi
{
source = %A_LoopFileFullPath%
destination = C:\out4\%A_LoopFileName%.mp4
Sleep 1000
run, "C:\Program Files (x86)\Handbrake\HandBrakeCLI.exe" -i "%source%" -o "%destination%" --encoder x264 -x264opts keyint=1 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0 , ,Hide , PID_HANDBREAK
Affinity_Set( 3, PID_HANDBREAK )
Sleep 300
Process, WaitClose, %PID_HANDBREAK%
if delsource = true
IfExist , %destination%
FileDelete %A_LoopFileFullPath%
}
goto top
this is the first functional version have made,
It polls my fraps location for new files, then send it to Handbreak CLI
after the encode it deletes the source file to ensure your fraps location does not fill up.
this creates aprox 150mb files from 4GB chunks however the color gets a bit messed but is corrected by the video editing suite. I need to do some more testing of different encode settings in handbreak but though this was worth sharing.

My purpose for this was I have been challenged to make a 100 hour YouTube video of my game play.
I'm a bit over half way there and still have more than sufficient space to continue collection on a single 500GB drive.

I'm keen to get some feedback about this. I have found a method of limiting handbreakCLI to a single thread permanently using ImageCFG.exe allowing my game to run smoothly during this process.

If there is any interest I will add a GUI for setting locations and encode quality.

My previous works if you care http://www.youtube.com/user/billman1987
 

raffriff

Moderator
Staff member
Site Contributor
Very interesting. I'd like to try it, but I know my system can't run a game, Fraps and Handbrake all at the same time.

What about the "file locking" issue, though? Doesn't Fraps prevent you from deleting any file while recording is going on? (While Fraps is recording, all files in the "session" (or group of 4GB files) are "locked" by the Fraps process. At least they were, last time I checked)

Other threads by people doing copying or re-encoding on the fly, which might interest you:
 

billman

Well-Known Member
Site Contributor
Thanks for your interest, I am using fraps version 3.5.9 and It indeed locks the files until it is finished, this is a good thing as my script will fail silently until the file is accessible, even if you select break into 4 GB chunks it will not allow you to remove the file until you stop capture, I am thinking about going back to an earlier version that is only capable of creating the 4GB chunks as it would finalise each video fly rather than wait until you stop recoding.

The RAM drive option is not viable for me as I tend to create up to 60 GB of video before Fraps realise the files.

Auto Copy is also not a good Idea as it will push the Fraps drive too hard If your trying to do full speed Write then a Full speed Read at the same time, even my RAID dose not handle this resulting in massive frame drop. With Handbreak or other command line encoding utilities cause heavy CPU usage (I limit to one core) but limit HDD usage. Resulting in about a 50/50 read/write scenario.

I know why I get frame drops, my windows raid only has so much cache and requires CPU (that is at 100 percent) to do work, If I used a real raid card this problem would go.

I see many other have this Idea seems I am one of the few pulling it off. My next problem is rendering over 24 hours in Sony Vegas seems to cause failure. It would seem I still have a few technical hurdles to get over but I will get there, I will have to browse this forum some more there is lots of different solutions here.
 

billman

Well-Known Member
Site Contributor
An update if anyone is following, I discovered HandbreakCLI while limited to one CPU is not quite fast enough to transcode at 30 FPS and at moments is causing almost a hard lock of the system for 5-30 seconds on seeming random occasions. Sony Vegas was causing the same thing when limited to one CPU. So at the moment my read/write rate is about 30/70 this has extended my max record time from 6 hours to almost 12. This would normally be enough but I'm not normal.

Next step is finding away to speed up Handbrake without degrading the footage, I am going to try and only allow Handbrake 2 CPU's, if that fail I guess I will have to use faster encode presets but at a cost of large output, almost defeating the purpose of what I'm doing.

Maybe I'm just dreaming, Recording a game, Transcoding Fraps Footage to MP4 and Rendering in Sony Vegas at once might just be a bit hard to do all at once, by god I will kill this computer trying.

Other interesting thing I found with Sony Vegas 11 is that the render seems to fail at about at about 14 hours thou the video or is failing once the output file hits about 64GB in size. So as it stands I am rendering 10 hour chunks each about 50 GB. That seems extreme but the output is beautiful considering 10 hours of fraps exceeds my 1.5 TB storage but I think I am going to need an additional HDD just to store the rendered output my OS drive is getting full fast.
 

raffriff

Moderator
Staff member
Site Contributor
Rendering files that large in Vegas, I think you would run into a RAM shortage problem.

With Handbrake or any x264 command line, use the "ultrafast" preset for maximum speed.

For screaming fast encodes with multicore processors, add "--keyint 1" to the command line (you might have to use the syntax variant "-x264opts keyint=1" with Handbrake). This makes each video frame stand on its own instead of being part of a large group (GOP). It allows each CPU core to work on a separate video frame, without waiting for other frames to finish processing. So depending on the number of cores, you may see another doubling of speed - or more.
https://encodingtalk.com/threads/fraps-to-youtube-with-h-264.894/#post-4756
It also vastly cuts RAM requirements. There is a file size penalty: you will have from 25-50% larger files.
 

billman

Well-Known Member
Site Contributor
Thank you for them suggestion on syntax for handbrake, I try that for my next round or recoding.
Sony Vegas top out at 2GB and Handbrake tops out 600MB, SAMP tops about 400MB of ram however I do seem to max about 6GB of "modified" RAM usage (files waiting to be written) and "Standby" eats the rest while this all this is going on at once. So I am getting close to using my 16GB.

I Set fraps back to Single File output and some of the Handbrake Transcodes are failing. Most notably if the resulting video file exceeds 4GB as the largest successful encode is 3.58 GB and 43 minutes long. So breaking Fraps output into 4GB chunks is my best bet for error free encoding or maybe another syntax option for handbrake would solve this.

EDIT: I found where I went wrong. If only I could read.

From the CLI guide itself

"-4 or --large-file if you do not enable it, and your MP4 is larger than 4 GB, the output will be unplayable and unrecoverable."


Some File Stats

Fraps Recordings - 222 GB
Resulted in 5.7GB failed encode




Successful Handbrake Output, - 3.58 GB



10 hour Sony Vegas output. - 50.6 GB
 

raffriff

Moderator
Staff member
Site Contributor
Okay, great. I'm surprised Vegas does so well. One thing, it's better to let Fraps make 4 GB segments IMO because if the computer crashes during recording you could lose your entire video versus losing only the final segment.
 

billman

Well-Known Member
Site Contributor
Processing FRAPS output directly was troublesome but these MP4's I'm using have been hassle free in Vegas (over 20 hours of footage processed without even reopening it) I used to save the project and reopen just to render previously, 4GB chunks are probably better however this computer has never crashed during recording despite the ass whooping I give it and having large continues files are easier to edit (sure you know that).

I think I solved the issues with the oversize MP4 with one switch, However now I have a drama with CPU affinity and handbrakeCLI.

I used IMAGECFG.exe to set it to Unicore to avoid it alone reducing the performance of the game while recording and undoing it should be a matter of replacing the modified EXE, but removing and reinstalling handbrake has not undone the affinity changes. I tested the same thing on handbrake.exe and replacing the EXE and doing so removed the affinity restrictions, but this is not true for CLI. WTF! 32bit complex or maybe cause I call it from Autohotkey this has me stumped.

I guess I will need to use "start /AFFINITY 3 handbrakeCLI.exe" (limit to CPU 0 and 1) however "WINWAIT" only waits until "START" has finished before running the next iteration of the AHK loop. This causes carnage (over 15 encodes at once w00t) so I can only foresee I have to make a more complex script to detect the presence of CLI and to pause the script before starting the next encode. Not hard I'm just lazy.

Also the the suggestion of single frame encoding plus a few other tweaks, took average encode FPS from 15 to 19 on single core, 25 to 40 on 2 Cores and now upto 50 FPS if allowed to have the whole CPU, File size chucks went from 150MB to 300MB for a 1 min 45 second video which is acceptable for me. This has also improved the quality of the output which I did not expect. I though faster encoding would produce substandard footage. I'm getting close to my goal.
 

billman

Well-Known Member
Site Contributor
That did the trick perfectly. Used Process, WaitClose, to pause the script after setting affinity with that method.
After 3 hours of recording I have produced 18.5 GB of transcoded footage, only 10 minutes of footage remained in the fraps folder after recording stopped. A few frame drops to tidy up and I'll be happy.

Thanks again.

Output Frames Looking GOOD
 

billman

Well-Known Member
Site Contributor
I rebuilt my bastard.ahk as the old one got stupid. This one has a gui for setting -
Source location
Output Location
Fraps.exe location
HandbrakeCLI.exe location

and has a go button, closing the GUI also stops the associated HandbrakeCLI.exe, checks to make sure the output file is a reasonable size, but I think my math is wrong here.

Code:
#SingleInstance force
Action = Startup
Gui, +Resize
Gui, Margin , 0, 0
Menu, FileMenu, Add, &Go,              Menu_Go
Menu, FileMenu, Add, &Source,          Menu_Source
Menu, FileMenu, Add, &Destination,  Menu_Destination
Menu, FileMenu, Add, &Fraps,          Menu_Fraps
Menu, FileMenu, Add, &Handbrake,      Menu_Handbrake
Menu, MyMenuBar, Add, &File,        :FileMenu
 
Menu, SettingsMenu, Add, &Fraps,          Settings_Fraps
Menu, SettingsMenu, Add, &Handbrake,      Settings_Handbrake
 
 
Menu, MyMenuBar, Add, &Settings,    :SettingsMenu
Gui, Menu, MyMenuBar
Gui, Add, ListView, xm h400 w700 vMyListView gMyListView , Time|Action|Output
Gui, Show , AutoSize gListView vListView, bastard
LV_Add("", A_Now, Action,A_ScriptFullPath)
gosub postmessage
 
gosub startup
return
 
Settings_Handbrake:
MsgBox comming soon
return
Settings_Fraps:
run "%fraps%"
return
 
 
Menu_Go:
IfEqual source
    gosub Menu_Source
IfEqual destination
    gosub Menu_Destination
IfEqual fraps
    gosub Menu_Fraps
IfEqual handbrake_CLI
    gosub Menu_Handbrake
gosub check_source
IfEqual handbrake_syntax
handbrake_syntax = --encoder x264 -x264opts keyint=1 -4 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0
return
 
Menu_Source:
FileSelectFolder, source , , 0 , Select Source Location
if errorlevel 1
return
Menu_Destination:
FileSelectFolder, destination , , 1 , Select Destination Location
return
Menu_Fraps:
;FileSelectFolder, fraps_dir , , 0 , Select Fraps.exe Location
fraps = C:\fraps\fraps.exe
IfNotExist fraps.exe
    goto Menu_Fraps
return
Menu_Handbrake:
FileSelectFolder, handbrake_dir , , 0 , Select HandbrakeCLI Location
handbrake_CLI = %handbrake_dir%\HandBrakeCLI.exe
IfNotExist %handbrake_dir%\HandBrakeCLI.exe
    goto Menu_Handbrake
return
 
transcode:
 
loop, parse, source_tobe_processed , `n ,
{
    Action = Transcode
    IfNotEqual, A_LoopField,
    {
    LV_Add("", A_Now, Action,A_LoopField)
    gosub postmessage
   
    Run %handbrake_CLI%  -i "%source%\%A_LoopField%" -o "%destination%\%A_LoopField%" %handbrake_syntax%,, Hide , handbrake_pid
   
    Process, wait, HandBrakeCLI.exe
    Results = HandbrakeCLI PID-%handbrake_PID%
    LV_Add("", A_Now, Action,Results)
    gosub postmessage
   
    Affinity_Set( 3, handbrake_pid )
    Process, WaitClose, HandBrakeCLI.exe
    Action = Complete   
    LV_Add("", A_Now, Action,A_LoopField)
    gosub postmessage
    gosub check_output
    gosub delete_source
    }
}
Action =
Sleep 10000
goto check_source
return
 
delete_source:
Action = Delete Source
if check_output_lock = locked
    return
IfExist , %destination%\%A_LoopField%
{
    FileGetSize, output_size , %destination%\%A_LoopField%, M
    FileGetSize, source_size , %source%\%A_LoopField%, M
    output_ratio = %source_size%
    output_ratio /= %output_size%
    Results = Input Video %source_size%MB Output Video %output_size%MB Reduced to %output_ratio% Percent.
    LV_Add("", A_Now, Action,Results)
    gosub postmessage
    if delete_source = true
        if output_ratio > 8
        {
            FileDelete , %source%\%A_LoopField%
            Results = %A_LoopField% Successful
            LV_Add("", A_Now, Action,Results)
            gosub postmessage
        }
        else
        {
            Results = %A_LoopField% Too Small Did Not Delete
            LV_Add("", A_Now, Action,Results)
            gosub postmessage
        }
}
Action =
return
 
 
source_tobe_processed:
Action = Processing
 
loop, parse, source_tobe_processed , `n ,
{
    IfNotEqual , source_tobe_processed
    {
        LV_Add("", A_Now, Action,A_LoopField)
        gosub postmessage
    }
}
Action =
goto transcode
return
 
check_output:
Action = Check Output
 
FileReadLine, check_output, %destination%\%A_LoopField%, 1
    if errorlevel = 0
    {
        check_output_lock = Unlocked
        LV_Add("", A_Now, Action,check_output_lock)
        gosub postmessage
    }
    else
    {
        check_output_lock = locked
        LV_Add("", A_Now, Action,check_output_lock)
        gosub postmessage
    }
return
 
 
check_source:
Action = Check Source
Loop, %source%\*.avi
{
    LV_Add("", A_Now, Action,A_LoopFileName)
    gosub postmessage
    FileReadLine, check_source, %A_LoopFileFullPath%, 1
    if errorlevel = 0
    {
        check_source_lock = Unlocked
        source_tobe_processed = %source_tobe_processed%`n%A_LoopFileName%
    }
    else
        check_source_lock = Locked
    LV_Add("", A_Now, check_source_lock,A_LoopFileName)
    gosub postmessage
}
Action =
 
goto source_tobe_processed
return
 
startup:
Action = Output Destination
delete_source = true
RegRead destination , HKCU, Software\Fraps3 , destination
LV_Add("", A_Now, Action,destination)
gosub postmessage
Action = HandbrakeCLI
RegRead handbrake_CLI , HKCU, Software\Fraps3 , handbrake_CLI
LV_Add("", A_Now, Action,handbrake_CLI)
gosub postmessage
RegRead handbrake_syntax , HKCU, Software\Fraps3 , handbrake_CLI_syntax
Action = Handbrake Syntax
handbrake_syntax = --encoder x264 -x264opts keyint=1 -4 --preset="Normal" -x ref=1:weightp=1:subq=0:rc-lookahead=10:trellis=0:8x8dct=0:cabac=0:me=dia:bframes=0
LV_Add("", A_Now, Action,handbrake_syntax)
gosub postmessage
Action = Fraps
RegRead fraps , HKCU, Software\Fraps3 , fraps
LV_Add("", A_Now, Action,fraps)
gosub postmessage
Action = Source Location
RegRead source, HKCU, Software\Fraps3 , source
LV_Add("", A_Now, Action,source)
gosub postmessage
Action =
 
return
 
F1::
RegWrite:
RegWrite, REG_SZ, HKCU, Software\Fraps3 , source, %source%
RegWrite, REG_SZ, HKCU, Software\Fraps3 , destination, %destination%
RegWrite, REG_SZ, HKCU, Software\Fraps3 , handbrake_CLI, %handbrake_CLI%
RegWrite, REG_SZ, HKCU, Software\Fraps3 , handbrake_syntax, %handbrake_syntax%
RegWrite, REG_SZ, HKCU, Software\Fraps3 , fraps, %fraps%
 
return
 
 
 
PostMessage:
PostMessage, 0x115 , 1, 1, SysListView321, bastard, ,,
LV_ModifyCol()
return
 
GuiSize:
    if A_EventInfo = 1
    {
    return
    }
    GuiControl, Move, MyListView, % "W" . (A_GuiWidth) . " H" . (A_GuiHeight )
return
 
MyListView:
if A_GuiEvent = DoubleClick
{
return
}
return
 
 
 
GuiClose:
gosub RegWrite
Process, Close, %handbrake_PID%
ExitApp
 
 
 
 
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Affinity_Set( CPU=1, PID=0x0 ) { ; CPU0=1 CPU1=2 | to use both, CPU should be 3
  Process, Exist, %PID%
  IfEqual,ErrorLevel,0,  SetEnv,PID,% DllCall( "GetCurrentProcessId" )
  hPr := DllCall( "OpenProcess",Int,1536,Int,0,Int,PID ) 
  DllCall( "GetProcessAffinityMask", Int,hPr, IntP,PAM, IntP,SAM )
  If ( CPU>0 && CPU<=SAM )
    Res := DllCall( "SetProcessAffinityMask", Int,hPr, Int,CPU )
  DllCall( "CloseHandle", Int,hPr )
Return ( Res="" ) ? 0 : Res
}
Finally starting to do some error handling but needs much more work yet.
But here is a screen for anyone following my progress.
 

raffriff

Moderator
Staff member
Site Contributor
Looks nice. I like a program with a clear & readable log file.

checks to make sure the output file is a reasonable size, but I think my math is wrong here.
you mean "if output_ratio > 8" ? If you are checking for a valid conversion, maybe you need to run MediaInfo CLI on both files and compare durations...
 

billman

Well-Known Member
Site Contributor
MediaInfo CLI sound like it should be the solution I was initially after. I was wondering how I could get the duration without writing my own binary reader or doing ridicules DLL calls, which is not all that hard to do just marjory time consuming. Looks like I might need an installer, or feature to grab Handbrake and MediaInfo from a remote source if not present. It was very early in the morning when I wrote the compare function and watching the output last night in the log proved it was bogus but still provide a baseline for me to work with. The log has been the biggest time consuming part of this script to make. But I am learning the fundamentals of state tracking and error handling, so it a bit buggy and wild but provides clear feedback on exactly what is going . Some more thought into the layout is needed but functionality take precedence over prettiness in my world.

I hope this makes sense to anyone but me, my brain is a bit fried I just finished a 15 hours gaming/recording stint and recorded about 90 percent of it, proving itself a very handy bit of script. However I have tracked my frame drop issue to the finalisation of the HandbrakeCLI encode. It would seem to me that having handbrakeCLI limited to 2 of 4 cores causes the CPU to be unable to do anything else during this time. I don't see any lockups when handbrake is give the whole CPU but I do struggle to record at 30FPS with this setting . On occasions it so bad the entire PC appears lockup for up to 5 seconds and shutter for up to 15 seconds. I'm guessing it is the L3 Cache with its token ring feature that enables peripherals to talk directly to the CPU cache. This cache is getting held up by the affinity setting. As each core requires access to the L3 cache disallowing the process to be pushed to the next core in the token ring it is physically halting the other cores until the current process is completed. This is only speculation of course but I can not find a spike in CPU time, HDD bandwidth, or excessive ram usage. Infact CPU usage appears to drop to zero during this time. Also and breaking the fraps output into 4gb chunks caused this to happen more frequently with slightly less severity. A FRAPS AVI of 300GB does not seem to prolong the shuttering significantly more than a 4GB FRAPS AVI .If I stop the recording during a shutter it take about 2 seconds from the time i hit the hotkey to the time my system comes back or I can allow it to record during the shutter and the audio plays and records fine (Game sound loops but streamed music is consistent). So far this has not caused any crashing or loss of connection to servers so that tells me the PCH (Platform Controller Hub) and peripherals is still ticking at normal rate during this event.
So an older Core 2 Quad may not have this issue. But it is the new cache system that provides cool features like the in chip video processor and intel quick sync and other hardware acceleration techniques that provide the biggest performance improvement over it predecessors .

These images should help you visualise how the Shared L3 Cache could cause it own bottleneck.


Above is a Quad-core Kentsfield package (2006)


Above is a Quad-core Sandy Bridge die (2011)

In my opinion forcing a quad core chip to talk to a single cache is a step backwards in terms of true parallelism but it benchmarks that sell chips not real world performance. Funny enough the old core 2 quads still stand up to todays games assuming you got some sweet ram and current video card. Enough of that.

I really like handbrake (took me ages to figure it out) but I now need to experiment with different command line based encoding solutions.

and raffriff thanks for taking a look at my script it is rather messy and I have already made some major changes to the order and way each feature is called, I don't feel quite so alone on my journey of technical discovery knowing that someone actually understand what I'm doing and why.
 

raffriff

Moderator
Staff member
Site Contributor
I'm guessing it is the L3 Cache with its token ring feature that enables peripherals to talk directly to the CPU cache. This cache is getting held up by the affinity setting. As each core requires access to the L3 cache disallowing the process to be pushed to the next core in the token ring it is physically halting the other cores until the current process is completed. This is only speculation of course but I can not find a spike in CPU time, HDD bandwidth, or excessive ram usage. In fact CPU usage appears to drop to zero during this time.
...
In my opinion forcing a quad core chip to talk to a single cache is a step backwards in terms of true parallelism...
Have never thought of that, but it seems logical. Brilliant!
 

billman

Well-Known Member
Site Contributor
Ok I have got a video to share, created but this process of encoding on the fly.
Frame quality is spectacular even once being uploaded to youtube.


Sony Vegas does not like the MP4's made by hand break making the editing a little harder than normal.
 
I would like to fraps on a ramdisk then recode to a hdd and delete the 4GB source file once recoded with handbrake.

Is it possible with this ahk ?
 
Top