ITimingTest

An Abstract base class that represents the tests that measure the median and time required to execute a well-defined operation. It is the main class used to measure CPU performance.

Instead of the usual setup, test, and cleanup framework methods, ITimingTest has timingSetup, timingTest, and timingCleanup framework methods. These methods behave somewhat differently from their namesakes. Subclasses place the code to be timed in timingTest. The methods timingSetup and timingCleanup allow subclasses to do any setup and cleanup for the timingTest method; code in these methods is NOT timed.

The ITimingTest base class enables you to write tests that measure the time taken by a specific operation to complete. The derived class IStartStopTimingTest allows you more control in timing events.

ITimingTest expects the operation to show some variation in execution time due to multitasking, varying CPU load, and other variable conditions. Consequently, the class takes multiple measurements and reports the median.

ITimingTest measures the time needed to execute the TimingTest member function. It does this by taking some number of samples, the "sample count". Each sample consists of a call to TimingSetup, some number of calls to TimingTest, and a call to TimingCleanup.

The calls to TimingTest happen in a tight loop, the "timing loop". Derived classes, or the caller through a RunTest option, can control how many iterations comprise the timing loop by changing the "timing count". The total time taken by the timing loop is divided by the number of iterations to get a single "sample time". The median of all sample times is taken as the estimate of the time to execute the TimingTest member function.

To create a timed test, do the following:

  1. Derive your test class from ITimingTest.
  2. Override TimingSetup, if necessary.
  3. Override TimingTest.
  4. Override TimingCleanup, if necessary.

The default settings are as follows:

     kDefaultTimingCount = 500,
     kDefaultSampleCount = 10,
     kDefaultMaximumSampleWindows = 10

kDefaultAutomaticSamplingTolerance = 0.1 kDefaultMinimumCorrelation = .4


ITimingTest - Member Functions and Data by Group

Constructors & Destructor

Class constructors and destructors.


[view class]
~ITimingTest
public:
virtual ~ITimingTest()
Destructor.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
ITimingTest


Overload 1
Standard copy constructor.
public:
ITimingTest(const ITimingTest& source)

source
ITimingTest to be copied.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Overload 2
protected:
ITimingTest( unsigned long timingCount, unsigned long sampleCount )
Constructor that explicitly sets timing count and sample count.

timingCount
explicit timing value.

sampleCount - explicit sample count value.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Overload 3
Default constructor.
protected:
ITimingTest()
This default constructor sets the timing count and sample count to default values, with no maximum allowed median time.

Constructs an ITimingTest specifying a timing count, sample count, maximum sample count, minimum correlation, and an automatic sample tolerance. The default values are:

     kDefaultTimingCount = 500,
     kDefaultSampleCount = 10,
     kDefaultMaximumSampleWindows = 10
     kDefaultAutomaticSamplingTolerance = 0.1
     kDefaultMinimumCorrelation = .4
     automatic sampling ON
     no maximum allowed median time

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Advanced Subclass Specialization Methods

Advanced timing member functions. These should only be used by those users with a good understanding of the timing tests.


[view class]
doUntimedTest
protected:
virtual void doUntimedTest()
Run the timingTest without performing the timing. (e.g., execute only timingSetup, timingTest, and timingCleanup). Useful to debug the timingTest without concern for the timing aspects.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
oneEmptySample
protected:
virtual double oneEmptySample()
Runs an empty sample. Useful to determine the amount of time necessary for the overhead of running timingTestt, the function call to an empty test function (e.g., timingTest return 0.0), and timingCleanup.

Return
theTime The time to run the empty sample

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
oneSample
protected:
virtual double oneSample()
Determine the time it takes to run one sample of the timingTest.

Return
theTime The time to run the one sample

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
samplingErrorMicroseconds
protected:
virtual double samplingErrorMicroseconds() const
Returns the sampling error.

Return
double The sampling error

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Assignment Operator

Use this operator to assign an ITimingTest class to another.


[view class]
operator =
public:
ITimingTest& operator =(const ITimingTest& rhs)
Standard assignment operator.

rhs
ITimingTest object used for assignment values.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Automatic Sampling Methods

Member functions used to configure the automatic timing sampling.


[view class]
automaticSampleTolerance
public:
double automaticSampleTolerance() const
Returns the threshold value for automatic sampling.

Return
double The current threshold value for automatic sampling

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
automaticSampling
public:
bool automaticSampling() const
Returns true if automatic sampling is done, false otherwise.

Return
bool State of automatic sampling setting

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
maximumSampleWindows
public:
unsigned long maximumSampleWindows() const
Returns the maximum number of sample windows.

Return
The maximum number of sample windows

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
minimumSampleCorrelation
public:
double minimumSampleCorrelation() const
Returns the minimum acceptable correlation between samples.

Return
The minimum acceptable correlation between samples

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setAutomaticSampleTolerance
public:
void setAutomaticSampleTolerance(double tolerance)
Sets the threshold value for automatic sampling. Samples will continue to be taken until the error on the measured mean is below this threshold value. For example, if the Tolerance is 0.05, then samples will continue to be taken until the error on the mean is less than or equal to 5 percent.

tolerance
The threshold value for automatic sampling

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setAutomaticSampling
public:
void setAutomaticSampling(bool doAutomaticSampling)
Controls whether automatic sampling is done.

doAutomaticSampling
If true, samples are teken until the error on the measured mean reaches a desired percentage, the tolerance. This allows the operation being timed to "settle" if the first measurements being taken are too high or low for some reason. Samples taken prior to settling are discarded. If false, then the first sampleCount samples are always taken and accepted.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setMaximumSampleWindows
public:
void setMaximumSampleWindows(unsigned long maxWindows)
Sets the maximum number of sample windows that are taken.

maxWindows
Number of sample windows to take

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setMinimumSampleCorrelation
public:
void setMinimumSampleCorrelation(double minSampleCorr)
Sets the minimum acceptable correlation between samples. If the correlation between samples exceeds this number, a message will be printed and the actual correlation will be posted.

minSampleCorr
The minimum correlation: 0 < minSampleCorr < 1

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
kDefaultAutomaticSamplingTolerance
public:
static const double kDefaultAutomaticSamplingTolerance
The default automatic sampling tolerance. Initial value set to 0.1.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
kDefaultMinimumCorrelation
public:
static const double kDefaultMinimumCorrelation
The default minimum correlation. Initial value set to 0.4.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Timing Methods

Class member functions used to control and perform the timing tests.


[view class]
calibrate
public:
void calibrate()
Measures the empty method time.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
calibratedMeanMicroseconds
public:
double calibratedMeanMicroseconds()
Returns the mean of all sampled times.

Return
double The mean of all sampled times

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
calibratedMedianMicroseconds
public:
double calibratedMedianMicroseconds()
Returns the estimated median time for the timingTest method to execute minus the empty method time, in microseconds.

Return
The estimated median time for the timingTest method

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
clockNowOverhead
public:
double clockNowOverhead() const
Returns the overhead required to read the clock. This is a measured value used to compute other error tolerances.

Return
double Overhead required to read the clock

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
continualCalibration
public:
bool continualCalibration() const
Returns true if calibrate is called for each sample; false otherwise.

Return
bool True if calibrate is called for each sample; false otherwise

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
copySamples
public:
void copySamples(long& count, double *&) const
This method sets a pointer to point at an ordered array of each sample value.

Note: The caller owns its storage for the elements in the array.

count
The number of elements of the array pointed to

double *& - the array pointer to set

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
correlationMicroseconds
public:
double correlationMicroseconds() const
Returns the correlation between samples.

Return
double The correlation between samples

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
correlationTooLarge
public:
bool correlationTooLarge() const
Returns true if the correlation between samples exceeds the minimum correlation specified by the user (using setMinimumSampleCorrelation).

Return
bool

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
errorMicroseconds
public:
double errorMicroseconds() const
Returns the 95% confidence interval for the test.

Return
double The 95% confidence interval for the test

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
errorTooLarge
public:
bool errorTooLarge() const
Returns true if the relative error from relativeErrorMicroseconds is greater than the tolerance specified by the tester on the command line or by calling setAutomaticSampleTolerance.

Return
bool

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
maximumAllowedMedianMicroseconds
public:
double maximumAllowedMedianMicroseconds() const
Returns the time limit for this timing test.

Return
double The time limit for this timing test.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
relativeErrorMicroseconds
public:
double relativeErrorMicroseconds() const
Returns the ratio of the 95% confidence interval for the mean (as returned by errorMicroseconds() ) to the mean itself.

Return
double The ratio of the 95% confidence interval

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
sampleCount
public:
unsigned long sampleCount() const
Returns the number of samples that will be taken for timing and calibration.

Return
long Number of samples that will be taken

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
samplesDiscarded
public:
unsigned long samplesDiscarded() const
Returns the number of samples discarded in the case that automatic timing is on.

Return
long The number of samples discarded

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
samplesTaken
public:
unsigned long samplesTaken() const
Returns the number of samples actually taken for a full test.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setContinualCalibration
public:
void setContinualCalibration(bool continuallyCalibrate)
Controls whether calibrate is called for each sample.

Continual calibration is off by default under the assumption that the empty method time is relatively constant. If the client or subclass is varying the load, then continual calibration may provide more accurate timing results.

continuallyCalibrate
bool; If true, timing is set to be continually calibrated

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setMaximumAllowedMedianMicroseconds
public:
void setMaximumAllowedMedianMicroseconds(double timeLimit)
Sets the time limit for this timing test. If the final calibrated median time exceeds the specified time limit, then the test is considered a failure and setSuccess(false) is called. If setMaximumAllowedMedianMicroseconds is never called (the default case) then no time limit will be set, and all runs will be reported to be passing tests.

timeLimit
Maximum allowed time

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setSampleCount
public:
void setSampleCount(unsigned long sampleCount)
Sets the number of samples that will be taken for timing and calibration. This is also the number of samples that will be taken (in a moving window) for automatic timing.

sampleCount
The number of samples to take

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setTimingCount
public:
void setTimingCount(unsigned long timingCount)
Sets the number of times the timingTest method will be called in a loop during a single sample.

Note: This timing count also applies to the calibration method.

timingCount
The number of calls to make within a single sample

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setTimingOn
public:
void setTimingOn(bool timingOn)
Controls whether timing operations are done for the timingTest method.

timingOn
A bool. If true, multiple samples are taken, each sample consisting of a call to timingSetup, many calls to timingTest, and a call to timingCleanup. If false, each run makes a single call to timingSetup, timingTest, and timingCleanup. In addition, echoing to the console is suppressed.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
timingCount
public:
unsigned long timingCount() const
Returns the number of times timingTest will be called in a loop during a single sample.

Return
long Number of times timingTest will be called

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
timingOn
public:
bool timingOn() const
Returns true if timing operations are done for the timingTest. Otherwise returns false.

Return
bool

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
cleanup
protected:
virtual void cleanup()
This is a framework cleanup method. It is called once after the test is run and should contain whatever code is necessary to cleanup after setup. It also must call the parent ITimingTest::cleanup method.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
print
protected:
virtual void print(ITieredTextBuffer& out)
Prints information about the object to the given tiered text buffer.

out
The buffer where information about the test is printed.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
setup
protected:
virtual void setup()
A framework setup method. It is called once before the test is run.

The test should override this function to execute any setup code that only needs to be called once.

It is also a good place to parse command line arguments. If this method is overridden it must call the parent ITimingTest::setup method.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
test
protected:
virtual void test()
A framework test method. ITimingTest overrides the ITest::test method; subtests of ITimingTest should not override it. ITimingTest::test first calls calibrate if this has not been done before. It then takes sampleCount timing samples if automatic sampling is off, or automatically takes enough samples to achieve the desired tolerance, as returned by automaticSamplingTolerance, up to a maximym of maximymSampleCount samples.

Note: Do NOT override this method!

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
timingCleanup
protected:
virtual void timingCleanup()
Performs any required cleanup for n consecutive calls to timingTest, where n is the timing count. Called by the framework once per sample. Subclasses must override timingCleanup if any cleanup is necessary after calls to timingTest.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
timingSetup
protected:
virtual void timingSetup()
Performs any required setup for n consecutive calls to timingTest, where n is the timing count. Called by the framework once per sample. Subclasses must override timingSetup if any setup is necessary before calls to timingTest.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


[view class]
timingTest
protected:
virtual void timingTest() = 0
A framework method for timing tests. Subclasses must override timingTest and place the code to be timed in the overridden method. If this code requires any setup and/or cleanup, subclasses should override timingSetup and timingCleanup and place the appropriate code in each.

For each timing sample, ITimingTest calls timingSetup once, followed by timingCount calls to timingTest, followed by a call to timingCleanup. The loop of calls to timingTest is timed, and this time is divided by timingCount to get the uncalibrated sample time. The median empty time is subtracted from this to get the calibrated sample time.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


Miscellaneous Members


[view class]
copyInfo
public:
virtual void copyInfo()
Adds information about input syntax and a timing test description. Can be overridden to provide metainformation.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


ITimingTest - Enumerations


[view class]
enum  { kDefaultTimingCount=500, 
        kDefaultSampleCount=10, 
        kDefaultMaximumSampleWindows=10 }
Default constants used by the timing tests.

Supported Platforms

Windows OS/2 AIX
Yes Yes Yes


ITimingTest - Inherited Member Functions and Data

Inherited Public Functions

ITest

Inherited Public Data

ITest

Inherited Protected Functions

ITest

Inherited Protected Data