This repository has been archived by the owner on Jun 19, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Small document describing the benchmark result pages and graphs.
- Loading branch information
Showing
1 changed file
with
65 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
# Index | ||
|
||
The | ||
[index page](http://storage.bos.xamarin.com/mono-gcbench/index.html) | ||
shows all configurations on which the benchmark suite was run. For | ||
the last revision, it shows the average benchmark runtime relative to | ||
the best performing revision so far, as well as the relative runtime | ||
of the worst-performing benchmark of the current revision. | ||
|
||
The configuration overview graphs are explained below. | ||
|
||
# Configuration page | ||
|
||
[Configuration pages](http://storage.bos.xamarin.com/mono-gcbench/default-sgen/index.html) | ||
summarize individual configuration results. The table has a row for | ||
each benchmark, giving times and revisions for the best and worst | ||
results, as well as absolute and relative times for the latest three | ||
revisions. | ||
|
||
It also shows small versions of the benchmark graphs for each | ||
benchmark. Those graphs are explained below. | ||
|
||
In the rightmost column it gives graphs of native code size, as | ||
reported by `mono --stats`. | ||
|
||
## Configuration overview graph | ||
|
||
In the | ||
[configuration overview graph](http://storage.bos.xamarin.com/mono-gcbench/default-sgen/combined_large.png) | ||
revision time moves from left to right, i.e. earlier revisions are | ||
left, later revisions are right. The vertical axis is relative | ||
runtime, so lower is better. | ||
|
||
The main (black) line shows the average relative benchmark runtime, | ||
i.e. it gives a good measure of how each revision performs on | ||
average. The green and red lines show the best and worst performing | ||
individual benchmark runtimes per revision, i.e. they indicate whether | ||
a revision has any outliers in terms of performance. | ||
|
||
The green and red markers indicate the best and worst performing | ||
revisions, respectively, based on average relative benchmark runtime. | ||
|
||
# Individual benchmark page | ||
|
||
Each individual | ||
[benchmark page](http://storage.bos.xamarin.com/mono-gcbench/default-sgen/graph4.html) | ||
lists results for all revisions the particular benchmark was run on | ||
the particular configuration. It presents average, minimum and | ||
maximum run times (in seconds), native code size, and gives a link to | ||
the raw run time files. | ||
|
||
## Benchmark graph | ||
|
||
In the | ||
[benchmark graph](http://storage.bos.xamarin.com/mono-gcbench/default-sgen/graph4_large.png), | ||
just as in the configuration overview graph, revision time moves from | ||
left to right and the vertical axis indicates runtime, in this case | ||
absolute. | ||
|
||
The main (black) line shows the average runtime, over three runs, | ||
whereas the gray area shows the variation of the runs, i.e. slowest to | ||
fastest. | ||
|
||
The green and red markers indicate the best and worst performing | ||
revisions for the benchmark, respectively. |