Test-Prioritization Tool

The test-prioritization tool enables the profile-guided optimizations on IA-32 and ItaniumŪ architectures to select and prioritize application's tests based on prior execution profiles of the application. The tool offers a potential of significant time saving in testing and developing large-scale applications where testing is the major bottleneck.

Development often requires changing applications modules. As applications change, developers can have a difficult time retaining the quality of their functional and performance tests so they are current and on-target. The test-prioritization tool lets software developers select and prioritize application tests as application profiles change.

Features and Benefits

The test-prioritization tool provides an effective testing hierarchy based on the code coverage for an application. The following list summarizes the advantages of using the tool:

See Understanding Profile-guided Optimization and Example of Profile-guided Optimization for general information on creating the files needed to run this tool.

Test-prioritization tool Requirements

The following items are files are required to run the test-prioritization tool on an applications tests:

Caution

The profmerge tool merges all .dyn files that exist in the given directory. Make sure unrelated .dyn files, which may remain from unrelated runs, are not present. Otherwise, the profile information will be skewed with invalid profile data, which can result in misleading coverage information and adverse performance of the optimized code

The tool uses the following general syntax:

Test-priorization tool Syntax

tselect -dpi_list file

where -dpi_list is a required tool option that sets the path to the list file containing the list of the all .dpi files.  All other commands are optional.

Note

Windows* only: Unlike the compiler options, which are preceded by forward slash ("/"), the tool options are preceded by a hyphen ("-").

Usage Model

The following figure illustrates the test-prioritization tool usage model.

Test-prioritization tool Options

The tool uses the options that are listed in the following table:

Option

Default

Description

-help

 

Prints options supported by the tool.

-spi file

pgopti.spi

Specifies the file name of the static profile information file (.SPI).

-dpi_list file

 

Specifies the file name of the file that contains the name of the dynamic profile information (.dpi) files.

Each line of the file must contain one .dpi name optionally followed by its execution time. The name must uniquely identify the test.

-prof_dpi dir

 

Specifies the path name of the output report file.

-o file

 

Specifies the file name of the output report file.

-comp file

 

Specifies the file name that contains the list of files of interest.

-cutoff value

 

Instructs the tool to terminate when the cumulative block coverage reaches a preset percentage, as specified by value , of pre-computed total coverage. value must be greater than 0.0 (for example, 99.00) but not greater than 100. value can be set to 100.

-nototal

 

Instructs the tool to ignore the pre-compute total coverage process.

-mintime

 

Instructs the tool to minimize testing execution time. The execution time of each test must be provided on the same line of dpi_list file, after the test name in dd:hh:mm:ss format.

-verbose

 

Instructs the tool to generate more logging information about program progress.

Running the tool

The following steps demonstrate one simple example for running the tool on IA-32 architectures.

  1. Specify the directory by entering a command similar to the following:

Example

set prof-dir=c:\myApp\prof-dir

  1. Compile the program and generate instrumented binary by issuing commands similar to the following:

Platform

Command

Linux

ifort -prof-genx myApp.f90

Windows

ifort /Qprof-genx myApp.f90

This commands shown above compiles the program and generates instrumented binary myApp as well as the corresponding static profile information pgopti.spi.

  1. Make sure that unrelated .dyn files are not present by issuing a command similar to the following:

Example

rm prof-dir \*.dyn

  1. Run the instrumented files by issuing a command similar to the following:

Example

myApp < data1

The command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by the -prof-dir step above.

  1. Merge all .dyn file into a single file by issuing a command similar to the following:

Example

profmerge -prof_dpi Test1.dpi

The profmerge tool merges all the .dyn files into one file (Test1.dpi) that represents the total profile information of the application on Test1.

  1. Again make sure there are no unrelated .dyn files present a second time by issuing a command similar to the following:

Example

rm prof-dir \*.dyn

  1. Run the instrumented application and generate one or more new dynamic profile information files that have an extension .dyn in the directory specified the prof-dir step above by issuing a command similar to the following:

Example

myApp < data2

  1. Merge all .dyn files into a single file, by issuing a command similar to the following

Example

profmerge -prof_dpi Test2.dpi

At this step, the profmerge tool merges all the .dyn files into one file (Test2.dpi) that represents the total profile information of the application on Test2.

  1. Make sure that there are no unrelated .dyn files present for the final time, by issuing a command similar to the following:

Example

rm prof-dir \*.dyn

  1. Run the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by -prof-dir by issuing a command similar to the following:

Example

myApp < data3

  1. Merge all .dyn file into a single file, by issuing a command similar to the following:

Example

profmerge -prof_dpi Test3.dpi

At this step, the profmerge tool merges all the .dyn files into one file (Test3.dpi) that represents the total profile information of the application on Test3.

  1. Create a file named tests_list with three lines. The first line contains Test1.dpi, the second line contains Test2.dpi, and the third line contains Test3.dpi.

Tool Usage Examples

When these items are available, the test-prioritization tool may be launched from the command line in prof-dir directory as described in the following examples.

Example 1: Minimizing the Number of Tests

The following example describes how minimize the number of test runs.

Example Syntax

tselect -dpi_list tests_list -spi pgopti.spi

where the -spi option specifies the path to the .spi file.

The following sample output shows typical results from the test-prioritization tool.

Sample Output

Total number of tests   =  3

Total block coverage    ~  52.17

Total function coverage ~  50.00

 

Num

%RatCvrg

%BlkCvrg

%FncCvrg

Test Name @ Options

1

87.50

45.65

37.50

Test3.dpi

2

100.00

52.17

50.00

Test2.dpi

In this example, the results provide the following information:

Example 2: Minimizing Execution Time

Suppose we have the following execution time of each test in the tests_list file:

Sample Output

Test1.dpi 00:00:60:35

Test2.dpi 00:00:10:15

Test3.dpi 00:00:30:45

The following command minimizes the execution time by passing the -mintime option:

Sample Syntax

tselect -dpi_list tests_list -spi pgopti.spi -mintime

The following sample output shows possible results:

Sample Output

Total number of tests   =  3

Total block coverage    ~  52.17

Total function coverage ~  50.00

Total execution time    =  1:41:35

 

Num

elapsedTime

%RatCvrg

%BlkCvrg

%FncCvrg

Test Name @ Options

1

10:15

75.00

39.13

25.00

Test2.dpi

2

41:00

100.00

52.17

50.00

Test3.dpi

In this case, the results indicate that the running all tests sequentially would require one hour, 45 minutes, and 35 seconds, while the selected tests would achieve the same total block coverage in only 41 minutes.  

The order of tests, when prioritization, is based on minimizing time (first Test2, then Test3) could be different than when prioritization is done based on minimizing the number of tests. See Example 1 shown above: first Test3, then Test2. In Example 2, Test2 is the test that gives the highest coverage per execution time, so Test2 is picked as the first test to run.

Using Other Options

The -cutoff option enables the tool to exit when it reaches a given level of basic block coverage. The following example demonstrates how to the option:

Example

tselect -dpi_list tests_list -spi pgopti.spi -cutoff 85.00

If the tool is run with the cutoff value of 85.00, as in the above example, only Test3 will be selected, as it achieves 45.65% block coverage, which corresponds to 87.50% of the total block coverage that is reached from all three tests.

The test-prioritization tool does an initial merging of all the profile information to figure out the total coverage that is obtained by running all the tests. The -nototal option enables you to skip this step. In such a case, only the absolute coverage information will be reported, as the overall coverage remains unknown.