mirror of https://github.com/CGAL/cgal
662 lines
29 KiB
TeX
662 lines
29 KiB
TeX
% \newcommand{\floor}[1]{\left\lfloor {#1} \right\rfloor}
|
|
% \newcommand{\unix}{{\tt UNIX}}
|
|
% \newcommand{\dos}{{\tt DOS}}
|
|
% \newcommand{\tildegen}{\protect\raisebox{-0.12cm}{\symbol{'176}}}
|
|
|
|
\section{Introduction}
|
|
\begin{verse}
|
|
``In the computer industry, there are three kinds of lies: lies, damn lies, and
|
|
benchmarks.''
|
|
\end{verse}
|
|
This document describes a practical toolkit to evaluate the status of code.
|
|
It can be used to create programs that measure performance, known as
|
|
benchmarks, and other various tests, execute them, and analyze their results.
|
|
With little effort a user of this toolkit can detect inefficiencies,
|
|
bottlenecks, and loss of functionality or performance degradation, compare
|
|
various techniques, algorithms, and different implementations, and measure
|
|
progress. A user, can then, present the results in a comprehensible way. The
|
|
information produced also includes the precise description of the execution
|
|
environment to allow the reproduction of the results.
|
|
|
|
There are two directives that must be followed in order to write bug-free code.
|
|
The first is to prevent the introduction of bugs in the first place. The second
|
|
is to find bugs as soon as they are introduced. This toolkit can be used to
|
|
build a full blown automatic system for bug detection, also known as regression
|
|
tests. We haven't used the toolkit in such a way so far, but we may exploit
|
|
this option in the future.
|
|
|
|
The toolkit is designed as a independent component in a large-scale
|
|
development environment. Final products and intermediate products that
|
|
are required to build other products are installed into a dedicated
|
|
database referred as {\em ROOT}, and pointed to by the environment
|
|
variable {\tt \$ROOT}. While the toolkit assumes that certain files are
|
|
installed into {\em ROOT}, it allows you to redefine {\em ROOT's}
|
|
location and other pieces that are based on {\em ROOT's} location
|
|
through command-line options.
|
|
|
|
The toolkit consists of three parts that corresponds to the three phases
|
|
required to evaluate code; (i) The creation of a hierarchy of test or benchmark
|
|
programs, (ii) the selective and controlled execution of the programs with
|
|
various input test cases, and (iii) the analysis and profiling of their results
|
|
and the conversion of the data into more meaningful and comprehensible
|
|
presentations. The following Sections describe the three parts in details.
|
|
|
|
It must be noted that our goal here is not to replace existing tools
|
|
that already provide useful functionality for computational
|
|
experiments (e.g., gnuplot, make, perl, python). Rather, the goal is
|
|
to augment this set with new tools that build on the functionality
|
|
already available to provide a comfortable testing environment.
|
|
|
|
It's worth mentioning the \textbf{ExpLab} tool set for Computational
|
|
Experiments by Susan Hert, Lutz Kettner, Tobias Polzin, and Guido
|
|
Sch{\"a}fer from the Max-Planck-Institut f{\"u}r Informatik. There seems to be
|
|
little overlap between \textbf{ExpLab} and our toolkit, but for most
|
|
they emphasize different aspects.
|
|
|
|
\section{Creation}
|
|
|
|
The toolkit was developed as a package in \cgal, the Computational
|
|
Geometry Algorithms and Data Structures library, to aid in the
|
|
development and maintenance of the planar map modules. As a
|
|
\cgal\ module on its own, it obliges to the Generic Programming Paradigm.
|
|
The first part consists of a couple of generic C++ classes, and the
|
|
interface between them and the user code to be evaluated. Some of the code
|
|
and documentation is derived from the material developed in the
|
|
\textsc{Acs} project, parts of which are published in \cite{cgal:f-ecsca-04}.
|
|
|
|
The \ccc{Bench_option_parser} class is used to parse command line options,
|
|
interpret them, and configure the bench accordingly. The benchmark itself
|
|
is performed through the \textbf{()} operator of the \ccc{Bench} class
|
|
described next.
|
|
|
|
The \ccc{Bench} class is parameterized with a model of the
|
|
{\em Benchable} concept. This concept serves as the interface between
|
|
the measuring device and the operation you wish to measure, refereed
|
|
as the target operation here after. A model of this concept must
|
|
satisfy a few requirements as follows. It must have a default
|
|
constructor, and it must provide four methods. The
|
|
\ccc{init()} method can be used to initialize some data
|
|
members before the target operation is carried out. The
|
|
\ccc{clean()} method is used to clean those data members
|
|
and residual data of the operation after the measure is completed. The
|
|
\ccc{sync()} method can be used to synchronize between the
|
|
target operation and the time-sampling operations. While these three
|
|
methods must be provided, they can be empty. The \ccc{op()}
|
|
method performs the target operation. This operation is executed in a
|
|
controlled loop according to various criteria explained in the next
|
|
paragraph.
|
|
|
|
If the estimated time-duration it takes to complete a single execution
|
|
of the target operation is large compared to the granularity of the
|
|
timing device, it is sufficient to execute the target operation a
|
|
small number of times between sampling the timing device, perhaps even
|
|
once. You can set the number of times the target operation is executed
|
|
through the \ccc{set_samples()} interface method of the
|
|
\ccc{Bench} class. On the other hand, if the estimated
|
|
time-duration of a single execution is of the same order as the
|
|
granularity of the timing device, increasing the number of executions
|
|
increases the accuracy of the measure. With the
|
|
\ccc{set_seconds()} method of the \ccc{Bench}
|
|
class, you can set the time slot in seconds allocated for the
|
|
measure. In this case the target operation is executed in a loop while
|
|
the number of executions is counted. When the time expires, the
|
|
counter is sampled. Then, the target operation is executed again for
|
|
as many times as counted, while measuring the time it takes to
|
|
complete the sequence.
|
|
|
|
\subsection*{A simple Benchmark Program}
|
|
The basic structure of a useful benchmark program can be very simple; Its
|
|
tasks are to measure the time it takes to complete a sequence of
|
|
executions of the target operation, and present the results in some useful
|
|
manner.
|
|
|
|
Below you can find the listing of a program that measures the time it takes
|
|
to compute the square root of $\Pi$, and produces the results shown in figure
|
|
\ref{sqrtResults}
|
|
|
|
\begin{figure}[!hbp]
|
|
\begin{verbatim}
|
|
Bench Bench Ops Total Single Num Ops
|
|
Name Time Num Ops Time Op Time Per Sec
|
|
-------------------------------- -------- -------- -------- -------- --------
|
|
Square root 1 14690512 1.0000 0.0000 14690512.0000
|
|
\end{verbatim}
|
|
\caption{Square-root benchmark results.}\label{sqrtResults}
|
|
\end{figure}
|
|
|
|
The \textbf{()} operator of the \ccc{Bench} class counts the number
|
|
of times the function \textbf{sqrt()} can be executed within the allocated time
|
|
slot. Then, it executes the \ccc{sqrt()} function for as many times as
|
|
counted, while measuring the time it takes to complete the sequence.
|
|
|
|
By default the allocated time slot is 1 second. The program overrides the
|
|
default with the number of seconds returned by the
|
|
\ccc{get_seconds()} method of the
|
|
\ccc{Bench_option_parser} class. The default of the later is
|
|
1 second as well, and can be overridden with the \textbf{``-t {\em seconds}''}
|
|
command-line option.
|
|
|
|
\ccIncludeExampleCode{Benchmark/simple.cpp}
|
|
|
|
\section{Bnechmark File Format}
|
|
We present a common file format for data instances and for benchmarking
|
|
\cgal\ software. It supports benchmarks on algebraic data as well as curve
|
|
and surface data. We document the file format grammar, a reference parser
|
|
implementation, and how it can be extended.
|
|
It is derived from the material developed in the \textsc{Acs} project.
|
|
|
|
\input{Benchmark/intro}
|
|
\input{Benchmark/benchmarkformat}
|
|
\input{Benchmark/grammar}
|
|
\input{Benchmark/visitor}
|
|
\input{Benchmark/extend}
|
|
|
|
\section{Execution}
|
|
This section lists the various command-line options a benchmark
|
|
program may accepts, and it explains how to create a hierarchy of
|
|
programs and execute the programs in the hierarchy selectively using
|
|
an agent implemented in \textbf{Perl}.
|
|
|
|
Some pieces of the toolkits are dedicated to the development and
|
|
maintenance of the \cgal\ planar map modules. For example, some of
|
|
the command-line options listed below directly control the behavior
|
|
of the planar-map benchmarks, and it's hard to imagine how they can be
|
|
applied to other benchmarks. While these pieces should be
|
|
reimplemented or even removed all together, in order to make the
|
|
toolkit fully generic, other users may ignore them for the time being.
|
|
|
|
\subsection{Command-Line Options}
|
|
A program written with the aid of the \ccc{Bench_option_parser}
|
|
class accepts the command-line options listed below. The command-line options
|
|
must be provided after the executable name and before an optional name of an
|
|
input file. A brief description is displayed on the console as a
|
|
response to the \textbf{`` -h''} option.
|
|
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abc}
|
|
}
|
|
\item[\bf{-b {\em options}}\hfill]
|
|
set bench options
|
|
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abc}
|
|
}
|
|
\item[\bf{type\_name={\em type}}\hfill]
|
|
\item[\bf{tn={\em type}}\hfill]
|
|
set bench type to {\em type} (default all).\\
|
|
{\em type} is one of:
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abc}
|
|
}
|
|
\item[\bf{i[ncrement]}\hfill] \bf{0x1}
|
|
\item[\bf{a[ggregate]}\hfill] \bf{0x2}
|
|
\item[\bf{d[isplay]}\hfill] \bf{0x4}
|
|
\end{list}
|
|
\item[\bf{type\_mask={\em mask}}\hfill]
|
|
\item[\bf{tm={\em mask}}\hfill]
|
|
set bench type mask to {\em mask}
|
|
\item[\bf{strategy\_name={\em strategy}}\hfill]
|
|
\item[\bf{sn={\em strategy}}\hfill]
|
|
set bench strategy to {\em strategy} (default all).\\
|
|
{\em strategy} is one of:
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abcdefg}
|
|
}
|
|
\item[\bf{t[rapezoidal]}\hfill] \bf{0x1}
|
|
\item[\bf{n[aive]}\hfill] \bf{0x2}
|
|
\item[\bf{w[alk]}\hfill] \bf{0x4}
|
|
\item[\bf{d[ummy]}\hfill] \bf{0x8}
|
|
\end{list}
|
|
\item[\bf{strategy\_mask={\em mask}}\hfill]
|
|
\item[\bf{sm={\em mask}}\hfill]\
|
|
set bench strategy mask to {\em mask}
|
|
\item[\bf{h[eader]={\em bool}}\hfill]
|
|
print header (default \textbf{true})
|
|
\item[\bf{name\_length={\em length}}\hfill]
|
|
\item[\bf{nl={\em length}}\hfill]
|
|
set the length of the name field to {\em length}
|
|
\end{list}
|
|
\item[\bf{-d {\em dir}}\hfill]
|
|
add directory {\em dir} to list of search directories
|
|
\item[\bf{-h}\hfill]
|
|
print this help message
|
|
\item[\bf{-i {\em iters}}\hfill]
|
|
set number of iterations to {\em iters} (default 0)
|
|
\item[\bf{-I {\em options}}\hfill]
|
|
set input options
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abcdefg}
|
|
}
|
|
\item[\bf{f[ormat]={\em format}}\hfill]
|
|
set format to {\em format} (default \textbf{rat}).\\
|
|
{\em format} is one of:
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abcdefg}
|
|
}
|
|
\item[\bf{i[nt]}\hfill] integer
|
|
\item[\bf{f[lt]}\hfill] floating point
|
|
\item[\bf{r[at]}\hfill] rational
|
|
\end{list}
|
|
\end{list}
|
|
\item[\bf{-r {\em root}}\hfill]
|
|
set the {\tt \$ROOT} to {\em root} (default is the environment
|
|
variable {\tt \$ROOT})
|
|
\item[\bf{-s {\em samples}}\hfill]
|
|
set number of samples to {\em samples} (default 10)
|
|
\item[\bf{-t {\em seconds}}\hfill]
|
|
set number of seconds to {\em seconds} (default 1)
|
|
\item[\bf{-v}\hfill]
|
|
toggle verbosity (default \textbf{false})
|
|
\end{list}
|
|
|
|
\subsubsection{Input}
|
|
The sole input file, if provided, must appear after the last
|
|
command-line option in the command line. This file is searched for in
|
|
a directory search-list. The initial list consists of the current
|
|
directory followed by {\tt \$ROOT/data/Segments\_2},
|
|
{\tt \$ROOT/data/Conics\_2}, and {\tt \$ROOT/data/Polylines\_2} in
|
|
this order, where {\tt \$ROOT} is initialized with the value of the
|
|
environment variable {\tt \$ROOT}, and possibly overridden using the
|
|
command line \textbf{``-r {\em root}''}
|
|
|
|
The \textbf{``-d {\em dir}''} command-line option inserts the
|
|
directory \textbf{\em dir} at the end of the search list.
|
|
|
|
We are aware to the need to extend the command-line parsing to handle
|
|
multiple input files. In addition the dedicated names
|
|
{\tt Segments\_2}, {\tt Conics\_2}, and {\tt Polylines\_2} should be
|
|
removed or extended.
|
|
|
|
The \ccc{get_input_format()} method of the
|
|
\ccc{Bench_option_parser} class returns the format provided
|
|
by the user through the \textbf{``-I format={\em format}''} command
|
|
line option, or \textbf{``-I f={\em format}''} in short.
|
|
|
|
\subsubsection{Output}
|
|
The output produced by a single benchmark program exemplified in
|
|
figure \ref{sqrtResults} is a text-based table easily readable by
|
|
humans. It consists of an optional header record and a data
|
|
record. The production of the header can be suppressed by the
|
|
\textbf{''-b header=false''} command-line option, or
|
|
\textbf{''-b h=false''} in short. Occasionally a sequence of
|
|
benchmarks are performed in a raw and the display of the header is
|
|
desired only once (or once per page).
|
|
|
|
A data record consists of the following fields:
|
|
\begin{description}
|
|
\item[Bench Name] the name of the benchmark.
|
|
\item[Bench Time] the allocated time-slot for the entire benchmark in
|
|
seconds (see \textbf{``-t {\em seconds}``} option).
|
|
\item[Ops Num] the number of target operations performed within the
|
|
time slot.
|
|
\item[Total Ops Time] the time required to perform the loop of
|
|
operations consisting of \textbf{Ops Num} operations in seconds.
|
|
\item[Single Op Time] the average time required to perform a single
|
|
operation in seconds.
|
|
\item[Num Ops Per Second] the number of operations completed per second.
|
|
\end{description}
|
|
|
|
The \textbf{Bench Name} field identifies the benchmark for all
|
|
purposes, its length is 32 characters by default. The length can
|
|
overridden by the \textbf{``-b name\_length={\em length}''}
|
|
command-line option, or \textbf{``-b nl={\em length}''} in short.
|
|
|
|
Independent tools listed in section \ref{Analysis} parse log files
|
|
that contain benchmark results, manipulate them, analyze them, and
|
|
perhaps convert them to other formats for artful presentations.
|
|
|
|
\subsection{Execution of multiple benchmarks}
|
|
The \textbf{Perl} script \textbf{cgal\_bench} selectively executes
|
|
multiple benchmarks ordered in a hierarchy. It executes them in a
|
|
sequence one at a time passing the appropriate command-line
|
|
options and input data file for each execution. It accepts a few
|
|
command-line options on its own listed below, and reads an input file
|
|
that contains the hierarchy of the benchmarks along with necessary
|
|
data required to execute them. This information is represented in a
|
|
simple language derived from the Extensible Markup Language (XML).
|
|
|
|
\subsubsection{Command-line options}
|
|
|
|
\begin{list}{}
|
|
{
|
|
\setlength{\topsep}{0pt}
|
|
\setlength{\partopsep}{0pt}
|
|
\setlength{\parskip}{0pt}
|
|
\setlength{\parsep}{0pt}
|
|
\setlength{\itemsep}{0pt}
|
|
\setlength{\itemindent}{0pt}
|
|
\setlength{\leftmargin}{0.2\textwidth}
|
|
\setlength{\labelsep}{0pt}
|
|
\setlength{\labelwidth}{0.2\textwidth}
|
|
\settowidth{\listparindent}{abc}
|
|
}
|
|
\item[\textbf{ -args {\em args}}\hfill]
|
|
set additional arguments passed to the benchmark programs.
|
|
\item[\textbf{-help}\hfill]
|
|
print this help message.
|
|
\item[\textbf{-verbose {\em level}}\hfill]
|
|
set verbose level to \textbf{\em level} (default 0).
|
|
\item[\textbf{-database {\em file}}\hfill]
|
|
set database xml file to \textbf{\em file} (default
|
|
{\tt \$ROOT/bench/data/benchDb.xml}).
|
|
\item[\textbf{-filter {\em name}}\hfill]
|
|
select bench \textbf{\em name}, and sub benches (default all).
|
|
\end{list}
|
|
|
|
A unique prefix is sufficient to indicate the desired
|
|
option. For example, when the \textbf{``-help''} option is specified,
|
|
a brief description is displayed on the console, and the script quits
|
|
immediately after. The same behavior is achieved through the
|
|
abbreviated \textbf{``-hel''}, \textbf{``-he''}, and \textbf{``-h''}
|
|
options.
|
|
|
|
By default the script reads the file
|
|
{\tt \$ROOT/bench/data/benchDb.xml}. This can be overridden through
|
|
the \textbf{``-database {\em file}''} command-line option.
|
|
|
|
\subsubsection{Documenting the Environment}
|
|
The scripts automatically documents the environment in which it
|
|
performs the benchmarks, so the benchmark can be easily rerun
|
|
(provided the same environment is still available) and the
|
|
results can be more accurately compared to the results of other
|
|
benchmarks.
|
|
|
|
The scripts extract most of the information directly from the
|
|
environment. Additional configuration data that cannot be extracted
|
|
directly from the environment is extracted from the database input
|
|
file. The script prints out its finding, and only then it starts
|
|
performing the benchmarks. Here is an excerpt from a sample run of
|
|
{\tt cgal\_bench}, showing the type of information extracted and
|
|
printed out.
|
|
|
|
\begin{verbatim}
|
|
Mon Mar 31 20:29:02 2003
|
|
COMPILER NAME: gcc
|
|
COMPILER INFO: gcc (GCC) 3.2
|
|
Copyright (C) 2002 Free Software Foundation, Inc.
|
|
This is free software; see the source for copying conditions. There is NO
|
|
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
|
|
|
|
OS NAME: linux
|
|
OS INFO: Linux cgal 2.4.20-net1 #1 SMP Wed Feb 5 13:05:52 IST 2003 i686 unknown
|
|
|
|
PROCESSOR: 0
|
|
CPU SPEED: 999.783 MHz
|
|
CPU TYPE: Pentium III (Coppermine)
|
|
PRIMARY DATA CACHE: 256 KB
|
|
SECONDARY DATA CACHE: 0
|
|
INSTRUCTION CACHE: 0
|
|
PROCESSOR: 1
|
|
CPU SPEED: 999.783 MHz
|
|
CPU TYPE: Pentium III (Coppermine)
|
|
PRIMARY DATA CACHE: 256 KB
|
|
SECONDARY DATA CACHE: 0
|
|
INSTRUCTION CACHE: 0
|
|
MEM SIZE: 2020 MBytes
|
|
GFX BOARD: unknown
|
|
CGAL VERSION: 2.5-I-81
|
|
LEDA VERSION: 441
|
|
QT VERSION: unknown
|
|
\end{verbatim}
|
|
|
|
\subsubsection{Input File Format}
|
|
The representation of the input file is derived from XML. Its element
|
|
tag-set consists of 4 predefined element tags listed below. All other
|
|
element tags that appear in an input file without exception are names
|
|
of executables that perform benchmarks.
|
|
|
|
The following is a list of the 4 elements with the 4 predefined tags
|
|
respectively:
|
|
\begin{description}
|
|
\item[file] specifies a file.
|
|
\item[bench] specifies a hierarchy of benchmarks.
|
|
\item[clo] specifies a command-line option.
|
|
\item[class] specifies a style-sheet class.
|
|
\end{description}
|
|
|
|
A \textbf{file} element specifies a data file provided as input to a
|
|
benchmark. It may have the following attributes:
|
|
\begin{description}
|
|
\item[name] - the file name.
|
|
\item[format] - the number type.
|
|
\item[curves] - the number of curves.
|
|
\item[vertices] - the number of vertices.
|
|
\item[halfedges] - the number of halfedges.
|
|
\item[faces] - the number of faces.
|
|
\end{description}
|
|
|
|
The \textbf{name} attribute is mandatory, as it identifies the file
|
|
for all purposes. The other attributes are optional (as a matter of
|
|
fact, the last 4 attributes are specific to the planar-map benchmarks.)
|
|
|
|
A \textbf{clo} element specifies a command-line option. It has the
|
|
following two mandatory attributes:
|
|
\begin{description}
|
|
\item[name] - the option name.
|
|
\item[string] - the option string.
|
|
\end{description}
|
|
|
|
The option name identifies the option for all purposes. The option
|
|
string is the exact argument that must appear in the command line for
|
|
that option to take effect.
|
|
|
|
A \textbf{bench} element specifies a hierarchy of benchmarks. It can
|
|
contain multiple \textbf{file}, \textbf{clo}, or \textbf{class}
|
|
elements, multiple elements that represent executables, and multiple
|
|
nested \textbf{bench} elements. The \textbf{file} attribute of a
|
|
\textbf{bench} element, if present, specifies an input data file. The
|
|
value of the \textbf{file} attribute is the name of the input file
|
|
(and the value of the \textbf{name} attributes of a \textbf{file}
|
|
element). Each attribute of a \textbf{bench} element that is neither
|
|
\textbf{file} nor \textbf{enable} specifies a command-line option. The
|
|
value of such an attribute is the option variable-value or
|
|
parameter. Command-line options are passed through inheritance in the
|
|
benchmark hierarchy, where an option parameter specified in a
|
|
\textbf{bench} element overrides the parameter specified higher in the
|
|
hierarchy. The boolean attribute \textbf{enable} simply indicates
|
|
whether the bench hierarchy should be executed or not.
|
|
|
|
The tag of a benchmark element is the name of an executable that
|
|
performs a benchmark. A benchmark element, just like a \textbf{bench}
|
|
element, may contain a \textbf{file} attribute to indicate an input
|
|
data file, a \textbf{enable} attribute to enable or disable the
|
|
benchmark, and multiple attributes, each indicating the parameter of a
|
|
command-line option.
|
|
|
|
A \textbf{class} element is used only while generating files for
|
|
browsing (e.g., html, php, etc.).
|
|
|
|
Figure \ref{database} lists a simple bench input-file that consists
|
|
of three benchmarks, three corresponding input files, and some
|
|
command-line options that are used to execute the benchmarks. When
|
|
this file is provided as input to the \textbf{cgal\_bench} script, the
|
|
later parses the file, interprets its contents, and executes the
|
|
commands below in turn:
|
|
\begin{verbatim}
|
|
bench1 -s 10 -bh=true -bnl=64 file1
|
|
bench2 -s 10 -bh=flase -bnl=64 file2
|
|
bench3 -s 10 -bh=flase -bnl=64 file3
|
|
\end{verbatim}
|
|
|
|
\begin{figure}[!hbp]
|
|
\begin{fminipage}{\textwidth}
|
|
\begin{alltt}
|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
|
|
<\textbf{\Red{bench}} name_length="64" header="false">
|
|
<\textbf{\GREEN{clo}} name="samples" string="-s"/>
|
|
<\textbf{\GREEN{clo}} name="name_length" string="-bnl="/>
|
|
<\textbf{\GREEN{clo}} name="header" string="-bh="/>
|
|
<\textbf{\GREEN{clo}} name="format" string="-If="/>
|
|
|
|
<\textbf{\GBLUE{file}} name="file1" format="rat"/>
|
|
<\textbf{\GBLUE{file}} name="file2" format="rat"/>
|
|
<\textbf{\GBLUE{file}} name="file3" format="rat"/>
|
|
|
|
<\textbf{\Red{bench}} samples="10" enable="true">
|
|
<\textbf{\Red{bench1}} file="file1" header="true"/>
|
|
<\textbf{\Maroon{bench2}} file="file2"/>
|
|
<\textbf{\Apricot{bench2}} file="file3"/>
|
|
</bench>
|
|
</bench>
|
|
\end{alltt}
|
|
\end{fminipage}
|
|
\caption{A simple database}\label{database}
|
|
\end{figure}
|
|
|
|
\section{Analysis\label{Analysis}}
|
|
The scripts in this category are intended to analyse and profile
|
|
results of benchmarks and convert the resulting data into more
|
|
meaningful and comprehensible presentations. They parse log files that
|
|
contain benchmark results, interpret the data, manipulate it, analyze
|
|
it, and perhaps convert it into other formats for artful presentations.
|
|
|
|
In principle they should be able to 1. add new fields to the
|
|
predefined fields of a benchmark data-record. This can be useful to
|
|
perform mathematical operations on the data values, or to reformat the
|
|
output for pretty printing, 2. merge records, 3. sort the records,
|
|
4. filter out records, and 5. convert the data into other formats.
|
|
|
|
Converting the textual results into other formats that support artful
|
|
presentations, pretty printing, and browsing capabilities is
|
|
deficient, as this part of the toolkit consists of a single
|
|
{\textbf Perl} script that converts benchmark results into a dedicated
|
|
php script that can be quickly integrated into the Tel-Aviv local
|
|
\cgal\ web site (not very useful for other users).
|
|
|
|
The \textbf{bash} command-line below can be used to record benchmark
|
|
results in separate log files in the {\tt \~{}/logs} directory:
|
|
|
|
\begin{verbatim}
|
|
cgal_bench <flags> 2>&1 | tee ~/logs/bcgal_`date +%y%m%d%%%H%M%S`.log
|
|
\end{verbatim}
|
|
|
|
When these log files are sorted by name, which is typically the
|
|
default, it is fairly easy to point at the most recent log file.
|
|
|
|
\section{Leftturn example}
|
|
The following example measures the performance of the left-turn predicate
|
|
in \cgal.
|
|
\ccIncludeExampleCode{Benchmark/leftturn.cpp}
|
|
|
|
The result
|
|
\begin{verbatim}
|
|
Bench Bench Ops Total Single Num Ops
|
|
Name Time Num Ops Time Op Time Per Sec
|
|
-------------------------------- -------- -------- -------- -------- --------
|
|
Leftturn 1 90242 0.9900 0.0000 91153.5345
|
|
\end{verbatim}
|
|
|
|
\section{Acknowledgement}
|
|
The benchmark file-format and the reference parser implementation was
|
|
developed at Max-Planck-Institut f\"ur Informatik, Saarbr\"ucken, Germany, by
|
|
Eric Berberich, Franziska Ebert, and Lutz Kettner as part of the
|
|
\textsc{Acs} project.
|
|
|
|
|
|
% conforms to the
|
|
% test agent
|
|
% performance profiling
|
|
% ------------------------------------------
|
|
% Automatic tests are arranged in a hierarchy. A set of super-tests comprises the highest level. Each super-test consists of a group of sub-tests that have something in common. For example, they all test X3D IndexedfaceSet features. Each sub-test is associated with an *.html file. The tests are executed sequentially. For each super-test, the browser is run from the command line. When the agent (script) that runs the browser, detects that the last sub-test within the current super has completed, it submits a kill signal that terminates the browser. For each sub-test the associated html file is loaded onto the browser from the command line. When the agent detects that the sub-test has terminated, it analyzes the test output to determine whether the test failed or succeeded. Next, the html associated with the next sub-test is loaded. These steps are repeated until all super-tests are run. The entire process is repeated for various configurations.
|
|
% ------------------------------------------
|
|
% 1 : the act or an instance of regressing
|
|
% 2 : a trend or shift toward a lower or less perfect state: as a :
|
|
% progressive decline of a manifestation of disease b (1) : gradual loss
|
|
% of differentiation and function by a body part especially as a
|
|
% physiological change accompanying aging (2) : gradual loss of memories
|
|
% and acquired skills c : reversion to an earlier mental or behavioral
|
|
% level d : a functional relationship between two or more correlated
|
|
% variables that is often empirically determined from data and is used
|
|
% especially to predict values of one variable when given values of the
|
|
% others <the regression of y on x is linear>; specifically : a function
|
|
% that yields the mean value of a random variable under the condition
|
|
% that one or more independent variables have specified values
|
|
% 3 : retrograde motion
|
|
% ------------------------------------------
|
|
% <benchmark> A standard program or set of programs which can be run on
|
|
% different computers to give an inaccurate measure of their
|
|
% performance.
|
|
%
|
|
% "In the computer industry, there are three kinds of lies: lies, damn
|
|
% lies, and benchmarks."
|
|
%
|
|
% A benchmark may attempt to indicate the overall power of a system by
|
|
% including a "typical" mixture of programs or it may attempt to measure
|
|
% more specific aspects of performance, like graphics, I/O or
|
|
% computation (integer or floating-point). Others measure specific tasks
|
|
% like rendering polygons, reading and writing files or performing
|
|
% operations on matrices. The most useful kind of benchmark is one which
|
|
% is tailored to a user's own typical tasks. While no one benchmark can
|
|
% fully characterize overall system performance, the results of a
|
|
% variety of realistic benchmarks can give valuable insight into
|
|
% expected real performance.
|
|
%
|
|
% Benchmarks should be carefully interpreted, you should know exactly
|
|
% which benchmark was run (name, version); exactly what configuration
|
|
% was it run on (CPU, memory, compiler options, single user/multi-user,
|
|
% peripherals, network); how does the benchmark relate to your workload?
|
|
%
|
|
% Well-known benchmarks include Whetstone, Dhrystone, Rhealstone (see
|
|
% h), the Gabriel benchmarks for Lisp, the SPECmark suite, and
|
|
% LINPACK.
|