Thank you. No manual Y-axis at this time, but it's a good idea! I'll add it sometime in the next few days if time permits.Very nice program, thanks for making this. Is there any way to manually set the Y-axis limits?
Exactly. Now that these charts are easier to make, I'd like to see them posted in all sorts of performance discussions.Now we need people to actually post benchmarks, especially after the techreport controversy regarding the 7950 latency problems.
Exploring “Frame time” measurement – Part 1 – Is Fraps a good tool? (alienbabeltech.com)
It has been noted for nearly a decade that just measuring fps does not convey the full experience if the frames are delivered unevenly - if there is stutter, micro stutter, or jitter. Recently, Scott Wasson of the Tech Report began to measure “frame times” which are also measured by Fraps. He is not the first, as Lost Circuits experimented with this in 2008 but returned to fps charts for simplicity sake.
It is pretty well-established that Fraps is a very good measure of frame rates (fps), but what about its ability to measure frame times? We are going to look at two in-game benchmarks that also measure frame times and we will compare them using Fraps frame time measurements.
...As you can see, Fraps reports very similarly to Metro 2033 in both frame rate and frame times.
Enter Frafs Bench Viewer
Lindsay Bigelow, an established member (raffriff) of FrapsForum.com, developed an uncomplicated tool for simply dragging and dropping Fraps frame time csv files into a program, named Frafs Bench Viewer, that quickly makes a chart. We would like to support his efforts by also using his tool and encouraging our readers to also make these simple charts and perhaps share them on ABT forum.
Frafs Bench Viewer can be downloaded at SourceForge.com or from Softpedia.com as freeware. All you do is drag and drop a Fraps frametimes.csv file into the program and it automatically creates customizable graphs!
The Frafs Benchmark Viewer is a very useful and easy-to-use tool to display frame times and we encourage our readers to submit their own graphs and experiences on ABT forum.
(Comment from DemBones79)
The first time I saw a frame latency (or whatever you're calling them now) graph, my first impression wasn't, "Wow, look at all these little latency spikes." It was, "Holy sh*t! Look at those huge freaking spikes!" It was a simple matter of severity. I think anyone can take a look at the "heartbeat", see that it is a recurring pattern with a relatively consistent frequency, and - while they may not be able to say for certain if it is indicative of a problem - they can say that it is "normal" for that particular card. It's the huge spikes, the ones that aren't occurring at consistent intervals, that are so much more severe than the "heartbeat", that are the issue.
How hard would it be for a reviewer to draw a pair of horizontal lines across the graph to indicate the limits of "normal" stuttering, where anything beyond the lines in either direction would be considered "abnormal"? A method of separating the signal from the noise.
That sums ups my feelings about the Anandtech article and one of the basic fundamental facts it managed not to see.http://www.anandtech.com/comments/6857/amd-stuttering-issues-driver-roadmap-fraps/296124
(Comment from DemBones79, continued)
Several sections in the article mention how FRAPS results may not be indicative of user experience. But it was user experience that prompted using FRAPS to try and explain what was being observed.
1) Smooth motion is achieved by having a consistent throughput of frames all the way from the game to the display.
2) Games measure the throughput of the pipeline via timing the back-pressure on the submission queue. The number they use to update their simulations is effectively what FRAPS measures as well.
3) A spike anywhere in the pipeline will cause the game to adjust the simulation time, which is pretty much guaranteed to produce jittery output. This is true even if frame delivery to the display (i.e. rendering pipeline output) remains buffered and consistent. i.e. it is never okay to see spikey output in frame latency graphs.
4) The converse is actually not true: seeing smooth FRAPS numbers does not guarantee you will see smooth display, as the pipeline could be producing output to the display at jittery intervals even if the input is consistent. This is far less likely though since GPUs typically do relatively simple, predictable work.
5) Measuring pipeline throughput (frames/second) over any multi-frame interval (entire runs, or even over a second like HardOCP) misses these issues because - as we saw above - throughput balances to the output of the pipeline when there are CPU spikes, so you'll have shorter frames after a long one. However frames that have had widely differing simulation time updates do not look smooth to the eye, even if they are delivered at a consistent rate to the display.
6) The problem that Scott most recently identified on AMD is likely a CPU driver problem, not a GPU issue (although the two can be related in some cases).
7) This is precisely why it is important to measure frame latencies instead of frame throughput. We need to keep GPU vendors, driver writers and game developers honest and focused on the task of delivering smooth, consistent frames. For example (unrelated to the above case), allowing things like a driver to cause a spike one frame so that it can optimize the shaders and get higher FPS number for the benchmarks is not okay!
8) If what we ultimately care about is smooth gameplay, gamers should be demanding frame latency measurements instead of throughput from all benchmarking sites.
Frame, Time (ms) 1, 0.000 2, 0.772 3, 2.631 4, 4.576 5, 5.368 6, 8.554 7, 9.265 8, 11.447 9, 13.356 10, 14.056
Frame, Time (ms) 1, 0.000 2, 33.337 3, 67.083 4, 100.201 5, 133.845 6, 167.045 7, 198.757 8, 233.548 9, 266.599 10, 299.808