Skip to content
This repository has been archived by the owner on Jun 19, 2024. It is now read-only.

xamarin/benchmarker

Folders and files

NameName
Last commit message
Last commit date
Oct 7, 2015
Oct 6, 2014
Aug 5, 2016
Jun 8, 2017
Aug 11, 2015
Jan 31, 2017
Mar 13, 2017
Jun 24, 2016
Dec 19, 2014
Jun 8, 2017
Aug 2, 2016
Nov 17, 2015
Apr 14, 2016
Nov 23, 2016
Aug 5, 2016
Mar 13, 2017
Mar 13, 2017
Aug 16, 2012
Nov 9, 2015

Repository files navigation

Mono Benchmarker

Build Status Go Report Card

Running benchmarks locally

Requirements

brew install lynx jq

Building

cd tools
nuget restore tools.sln
xbuild /target:compare

Running

There are two ways to run the benchmarking suite locally, one higher-level than the other. If you want to benchmark finished Mono packages from a build bot, you should use run.sh. If you want to benchmark a Mono that you've built on your own machine, use the lower-level compare.exe.

From a package

tools/run.sh requires at least three pieces of information, two of which serve as documentation and for reproducibility:

  • The file or URL of the Mono package you want to benchmark
  • The URL of the build page for that package
  • The mono Git repository commit of that package

Example, on OS X:

./tools/run.sh --commit 2d4f6b344205b410f21a3470d4152d78a28964d1 \
               --build-url 'https://wrench.internalx.com/Wrench/ViewLane.aspx?lane_id=1972&host_id=148&revision_id=580021' \
               --pkg-url 'http://storage.bos.internalx.com/mono-mac-4.2.0-pre2-branch/2d/2d4f6b344205b410f21a3470d4152d78a28964d1/MonoFramework-MRE-4.2.0.22.macos10.xamarin.x86.pkg'

It will ask for your password when it tries to install the package. The package is installed in a temporary disk image, so your system will not be affected.

On Linux, run.sh requires that you specify .deb package URLs. --help will tell you how.

From a local build

To run the suite on a locally built Mono, use tools/compare.exe:

mono tools/compare.exe --root <MONO-INSTALL-ROOT>

Your Mono executable must be in MONO-INSTALL-ROOT/bin/mono-sgen. The benchmark runner will try to figure out the Git commit of your Mono. If that fails, use the --commit and --git-repo options to help it.

Once all benchmarks have been run and the results uploaded it will print a URL that shows all the results.

Setting options

To give options to Mono or set environment variables you need to use a configuration files. Those files are kept in the config directory. The default configuration is default-sgen.conf. Make a copy of it and rename it, to avoid confusion and for documentation.

Configs

Each Mono configuration requires a .conf file. The files in the configs directory are examples. The JSON structure is as follows:

  • Name: name of the config (must be unique across all configs and benchmarks)
  • Count: number of time to run the benchmark (optional, default: 10)
  • Mono: path to the mono executable (optional, default to system one)
  • MonoOptions: command line parameters to pass to the mono runtime (optional)
  • MonoEnvironmentVariables: environment variables to set to run the benchmark (optional)
  • UnsavedMonoEnvironmentVariables: environment variables to set to run the benchmarks, but not saved in the database (optional)

Benchmarks

Each benchmark requires a .benchmark file. The files in the benchmarks directory are examples. The JSON structure is as follows:

  • Name: name of the benchmark (must be unique across all configs and benchmarks)
  • TestDirectory: path to the working directory to run the benchmark, relative to the benchmarker repository root directory
  • CommandLine: command line to run the benchnark, does not contain the runtime (mono) executable
  • Timeout: benchmark specific timeout, override the command line one

Building the front-end

Requirements

brew install npm
npm -g install webpack

Building

make -C front-end