Frafs Bench Viewer - tool for viewing Fraps benchmark files

Thanks for the well designed app. Now we need people to actually post benchmarks, especially after the techreport controversy regarding the 7950 latency problems. I'd personaly like to see some 7950 vs GTX 660Ti graphs posted here.
 

raffriff

Moderator
Staff member
Site Contributor
Very nice program, thanks for making this. Is there any way to manually set the Y-axis limits?
Thank you. No manual Y-axis at this time, but it's a good idea! I'll add it sometime in the next few days if time permits.

Now we need people to actually post benchmarks, especially after the techreport controversy regarding the 7950 latency problems.
Exactly. Now that these charts are easier to make, I'd like to see them posted in all sorts of performance discussions.
 

raffriff

Moderator
Staff member
Site Contributor
Thanks Plonk...there's another mention on alienbabeltech.com:
Exploring “Frame time” measurement – Part 1 – Is Fraps a good tool? (alienbabeltech.com)
It has been noted for nearly a decade that just measuring fps does not convey the full experience if the frames are delivered unevenly - if there is stutter, micro stutter, or jitter. Recently, Scott Wasson of the Tech Report began to measure “frame times” which are also measured by Fraps. He is not the first, as Lost Circuits experimented with this in 2008 but returned to fps charts for simplicity sake.

It is pretty well-established that Fraps is a very good measure of frame rates (fps), but what about its ability to measure frame times? We are going to look at two in-game benchmarks that also measure frame times and we will compare them using Fraps frame time measurements.

...As you can see, Fraps reports very similarly to Metro 2033 in both frame rate and frame times.

Enter Frafs Bench Viewer
Lindsay Bigelow, an established member (raffriff) of FrapsForum.com, developed an uncomplicated tool for simply dragging and dropping Fraps frame time csv files into a program, named Frafs Bench Viewer, that quickly makes a chart. We would like to support his efforts by also using his tool and encouraging our readers to also make these simple charts and perhaps share them on ABT forum.

Frafs Bench Viewer can be downloaded at SourceForge.com or from Softpedia.com as freeware. All you do is drag and drop a Fraps frametimes.csv file into the program and it automatically creates customizable graphs!

The Frafs Benchmark Viewer is a very useful and easy-to-use tool to display frame times and we encourage our readers to submit their own graphs and experiences on ABT forum.
:D
 
Nice Tool. But it's missing two functions!

"Export"
Press one button saves "Time" "FPS" and "Rank" (or maybe even only thos chosen in settings, me myself i don't need fps) as images into a certain(chosen in settings) folder.

"Export to FTP"
Same as Export just that you can configure a FTP server(folder) in advance where you want to save them.
~ this might be complicated because you'd need to integrate a FTP client into this tool and then you might get problems with Firewall a.m.m. ... But i'd be kinda cool for online sharing.
Maybe something like an direct upload to imgbanan, imgur, imageshack also...


But that might only be important to me and not for anyone else.
 

raffriff

Moderator
Staff member
Site Contributor
Thanks for your input, it's always welcome. "Export" sounds like a good idea; it'll be in the next update - when that will be, I'm not sure.

Remote file transfer seems like a lot of work though, TBH. Use the FireFTP plugin for Firefox, imgur upload tools, etc, which work better than any tool I could write.
 
Great tool!

The Avg label is (sometimes) overlapping another label for the Y axis.
1% and 0.1% times are shown, but 0% isn't, might be useful.
I've no idea what Half size and Unlocked checkboxes do. A tooltip would be handy.
When zooming, it'd be nice if the X position under the pointer stayed constant.

An option to use dots instead of bars for the graph could be useful for wildly fluctuating frametimes.
 

raffriff

Moderator
Staff member
Site Contributor
Good suggestions; thank you.

The Half size and Unlocked checkboxes, along with resolution and frame rate, are only for the annotation text. Tool tips are a good idea, as that's not very clear.

0% times wouldn't mean anything [if I understand you correctly], unless you want the very worst frame time of the test run. That's pretty easy to get now, by selecting the "Rank" view, but I may add it to the annotations as well. [You can also check the Min/Max/Average box in Fraps]
 
I think the very worst frame time isn't meaningless at all.

One other issue: the graph tooltip flickers.
 

raffriff

Moderator
Staff member
Site Contributor
OK, I'll add the best, worst and average times.
I thought I fixed the tooltip flicker, but I'll take another look at at.
 

raffriff

Moderator
Staff member
Site Contributor
Thanks Yanak, they do look interesting. I will study them as soon as I can.
 

raffriff

Moderator
Staff member
Site Contributor
http://www.anandtech.com/comments/6857/amd-stuttering-issues-driver-roadmap-fraps/296124
(Comment from DemBones79)

The first time I saw a frame latency (or whatever you're calling them now) graph, my first impression wasn't, "Wow, look at all these little latency spikes." It was, "Holy sh*t! Look at those huge freaking spikes!" It was a simple matter of severity. I think anyone can take a look at the "heartbeat", see that it is a recurring pattern with a relatively consistent frequency, and - while they may not be able to say for certain if it is indicative of a problem - they can say that it is "normal" for that particular card. It's the huge spikes, the ones that aren't occurring at consistent intervals, that are so much more severe than the "heartbeat", that are the issue.

How hard would it be for a reviewer to draw a pair of horizontal lines across the graph to indicate the limits of "normal" stuttering, where anything beyond the lines in either direction would be considered "abnormal"? A method of separating the signal from the noise.
Hear hear.

http://www.anandtech.com/comments/6857/amd-stuttering-issues-driver-roadmap-fraps/296124
(Comment from DemBones79, continued)

Several sections in the article mention how FRAPS results may not be indicative of user experience. But it was user experience that prompted using FRAPS to try and explain what was being observed.
That sums ups my feelings about the Anandtech article and one of the basic fundamental facts it managed not to see.

--------

Meanwhile, PC Perspective's article was much more objective and informative in my opinion. The article pointed to a very good post on beyond3d by an Intel GPU engineer discussing microstutter which seems to say that Fraps is a valid tool for uncovering problems.
http://forum.beyond3d.com/showthread.php?p=1689708

1) Smooth motion is achieved by having a consistent throughput of frames all the way from the game to the display.

2) Games measure the throughput of the pipeline via timing the back-pressure on the submission queue. The number they use to update their simulations is effectively what FRAPS measures as well.

3) A spike anywhere in the pipeline will cause the game to adjust the simulation time, which is pretty much guaranteed to produce jittery output. This is true even if frame delivery to the display (i.e. rendering pipeline output) remains buffered and consistent. i.e. it is never okay to see spikey output in frame latency graphs.

4) The converse is actually not true: seeing smooth FRAPS numbers does not guarantee you will see smooth display, as the pipeline could be producing output to the display at jittery intervals even if the input is consistent. This is far less likely though since GPUs typically do relatively simple, predictable work.

5) Measuring pipeline throughput (frames/second) over any multi-frame interval (entire runs, or even over a second like HardOCP) misses these issues because - as we saw above - throughput balances to the output of the pipeline when there are CPU spikes, so you'll have shorter frames after a long one. However frames that have had widely differing simulation time updates do not look smooth to the eye, even if they are delivered at a consistent rate to the display.

6) The problem that Scott most recently identified on AMD is likely a CPU driver problem, not a GPU issue (although the two can be related in some cases).

7) This is precisely why it is important to measure frame latencies instead of frame throughput. We need to keep GPU vendors, driver writers and game developers honest and focused on the task of delivering smooth, consistent frames. For example (unrelated to the above case), allowing things like a driver to cause a spike one frame so that it can optimize the shaders and get higher FPS number for the benchmarks is not okay!

8) If what we ultimately care about is smooth gameplay, gamers should be demanding frame latency measurements instead of throughput from all benchmarking sites.
 
Yup but main thing is that Fraps is not reliable to determine real frame latency, it can help to see there is a problem but after this you need better tools to get the last word and as we see in some tests with crossfire even for the FPS indicated it's sometimes false, well i'm waiting for more reviews that should come soon about all this and hopefully AMD will bring some good news in July like they said for last gen cards by fix stuttering in games with some new drivers.
 

billman

Well-Known Member
Site Contributor
Yanak have you ever read a fraps benchmark file?

Here is an example of 120 FPS
Code:
Frame, Time (ms)
    1,    0.000
    2,    0.772
    3,    2.631
    4,    4.576
    5,    5.368
    6,    8.554
    7,    9.265
    8,    11.447
    9,    13.356
  10,    14.056
Here is an example of 30 FPS
Code:
Frame, Time (ms)
    1,    0.000
    2,    33.337
    3,    67.083
    4,  100.201
    5,  133.845
    6,  167.045
    7,  198.757
    8,  233.548
    9,  266.599
  10,  299.808
That looks like and accurate list of frames and how long it took to draw each one down to one thousandth of a millisecond. In human terms that is 1 million times per second.
 
? :confused:

Yes i read them since way before this latency thing came up while in some alpha/beta testing games programs to try to narrow down and understand some issues they had and done my own graphics using open office since years, and so i return you the favor , did you read some articles explaining why Fraps is not accurate at reading the FPS because it does it at one moment of the process that is not correct to get accurate results and those results might be way different than the reality displayed on your screen especially in CFX configurations ???

here for example :
http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-Dissected-Full-Details-Capture-based-Graphics-Performance-Testin ?
 

billman

Well-Known Member
Site Contributor
I had read that article and watched the videos to be sure I understood what they were doing. Honestly I appreciate what these guys are doing but these methods involve too many possible variables. Any sort of capture method has it own set of issues and the fact they had to lower the resolution of their capture card tells me there capture hardware is not up to doing what they're trying to achieve. The only method I can perceive that is more accurate than measuring pipeline performance is to connect an oscilloscope to the relevant output of the graphics engine itself. Personally I find fraps counter to be misleading however the benchmark files are very specific.

The reason these guys get different results to fraps is because the act of measuring the pipeline will have a negative effect on framerate, as where using an external capture device has no such impact but can only analyse the visible frames that made it to the output. On occasion a frame can be so short it is never actually drawn or the PC doing the capture may not be fast enough or have its own stuttering issues.
Its good to see people trying to devise methods like this but they make me laugh so hard.
 

raffriff

Moderator
Staff member
Site Contributor
Bench Viewer 0.2.8.3. Lots of small enhancements.
  • Improved zoom on Y axis (can shrink as well as grow)
  • Redesigned the 'Edit User Info' dialog box for better clarity.
  • Chart shows the true 'data point' of each frame more clearly.
  • Fixed frame info tooltip 'flickering' when near edge of screen.
  • Easier to pick out very high & low values with Frame tooltip.
  • X axis labeled in 'real' time, i.e. time-of-day.
  • No more 30,000 frame limit; new limit = 8.6 million frames,
    but 'only' 360,000 can be shown at one time; pan with the
    mouse to see the rest of your data.
  • In general, better handling of very large benchmark files.
 
Top