Skip navigation links
The MySQL Test Framework, Version 2.0 :: 2 MySQL Test Framework Components
2.1 Test Framework System Requirements »
Section Navigation      [Toggle]

Chapter 2. MySQL Test Framework Components

Table of Contents     [+/-]

2.1. Test Framework System Requirements
2.2. The Test Framework and SSL
2.3. How to Report Bugs from Tests in the MySQL Test Suite

The MySQL test framework consists of programs that run tests, and directories and files used by those programs.

Test Framework Programs

The MySQL test framework uses several programs:

  • The mysql-test-run.pl Perl script is the main application used to run the test suite. It invokes mysqltest to run individual test cases.

  • mysqltest runs test cases. A version named mysqltest_embedded is similar but is built with support for the libmysqld embedded server.

  • The mysql_client_test program is used for testing aspects of the MySQL client API that cannot be tested using mysqltest and its test language. mysql_client_test_embedded is similar but used for testing the embedded server.

  • The mysql-stress-test.pl Perl script performs stress-testing of the MySQL server.

  • A unit-testing facility is provided so that individual unit test programs can be created for storage engines and plugins.

Test suite programs can be found in these locations:

  • For a source distribution, mysqltest is in the client directory. For a binary distribution, it is in the MySQL bin directory.

  • For a source distribution, mysql_client_test is in the tests directory. For a binary distribution, it is in the MySQL bin directory.

  • The other programs are located in the mysql-test directory. For a source distribution, mysql-test is found under the source tree root. For a binary distribution, the location of mysql-test depends on the layout used for the distribution format.

Test Framework Directories and Files

The test suite is located in the mysql-test directory, which contains the following components:

  • The mysql-test-run.pl and mysql-stress-test.pl programs that are used for running tests.

  • The t directory contains test case input files. A test case file might also have option files associated with it.

    • A file name of the form test_name.test is a test case file for a test named test_name. For example, subquery.test is the test case file for the test named subquery.

    • A file name of the form test_name-master.opt provides options to associate with the named test case. mysql-test-run.pl restarts the server with the options given in the file if the options are different from those required for the currently running server.

      Note that the -master.opt file is used for the “main” server of a test, even if no replication is involved.

    • A file name of the form test_name-slave.opt provides slave options.

    • A file name of the form test_name-master.sh is a shell script that will be executed before the server is started for the named test case. There may also be a test_name-slave.sh which is executed before the slave server is started.

      These files will not work on Windows and may therefore be replaced with a more portable mechanism in the future.

    • The disabled.def file contains information about deferred/disabled tests. When a test is failing because of a bug in the server and you want it to be ignored by mysql-test-run.pl, list the test in this file.

      The format of a line in the disabled.def file looks like this, where fields are separated by one or more spaces or Tab characters: 

      test_name : BUG#nnnnn YYYY-MM-DD disabler comment

      Example: 

      rpl_row_blob_innodb : Bug#18980 2006-04-10 kent  Test fails randomly

      test_name is the test case name. BUG#nnnnn indicates the bug related to the test that causes it to fail (and thus requires it to be disabled). disabler is the name of the person that disabled the test. comment normally provides a reason why the test was disabled.

      The text following the colon is not part of the mandatory syntax for this file; mysql-test-run.pl will not actually care what is written here. However, it is a recommended standard.

      A comment line can be written in the file by beginning the line with a “#” character.

  • The r directory contains test case result files:

    • A file name of the form test_name.result is the expected result for the named test case. A file r/test_name.result is the output that corresponds to the input in the test case file t/test_name.test.

    • A file name of the form test_name.reject contains output for the named test case if the test fails due to output mismatch (but not it it fails for other reasons).

    For a test case that succeeds, the .result file represents both the expected and actual result. For a test case that fails, the .result file represents the expected result, and the .reject file represents the actual result.

    If a .reject file is created because a test fails, mysql-test-run.pl removes the file later the next time the test succeeds.

  • The include directory contains files that are included by test case files using the source command. These include files encapsulate operations of varying complexity into a single file so that you can perform the operations in a single step. See Section 4.11, “Using Include Files to Simplify Test Cases”.

  • The lib directory contains library files used by mysql-test-run.pl, and database initialization SQL code.

  • The std_data directory contains data files used by some of the tests.

  • The var directory is used during test runs for various kinds of files: log files, temporary files, trace files, Unix socket files for the servers started during the tests, and so forth. This directory cannot be shared by simultaneous test runs.

  • The suite directory contains a set of subdirectories containing named test suites. Each subdirectory represents a test suite with the same name.

  • The collections directory contains collections of test runs that are run during integration and release testing of MySQL. They are not directly useful outside this context, but need to be part of the source repository and are included for reference.

Unit test-related files are located in the unittest directory. Additional files specific to storage engines and plugins may be present under the subdirectories of the storage or plugin directories.

Test Execution and Evaluation

There are a number of targets in the top-level Makefile that can be used to run sets of tests. make test runs all the tests. Other targets run subsets of the tests, or run tests with specific options for the test programs. Have a look at the Makefile to see what targets are available.

A “test case” is a single file. The case might contain multiple individual test commands. If any individual command fails, the entire test case is considered to fail. Note that “fail” means “does not produce the expected result.” It does not necessarily mean “executes without error,” because some tests are written precisely to verify that an illegal statement does in fact produce an error. In such an instance, if the statement executes successfully without producing the expected error, that is considered failure of the test.

Test case output (the test result) consists of:

  • Input SQL statements and their output. Each statement is written to the result followed by its output. Columns in output resulting from SQL statements are separated by tab characters.

  • The result from mysqltest commands such as echo and exec. The commands themselves are not echoed to the result, only their output.

The disable_query_log and enable_query_log commands control logging of input SQL statements. The disable_result_log and enable_result_log commands control logging of SQL statement results, and warning or error messages resulting from those statements.

mysqltest reads a test case file from its standard input by default. The --test-file or -x option can be given to name a test case file explicitly.

mysqltest writes test case output to the standard output by default. The --result-file or -R option can be used to indicate the location of the result file. That option, together with the --record option, determine how mysqltest treats the test actual and expected results for a test case:

  • If the test produces no results, mysqltest exits with an error message to that effect, unless --result-file is given and this file is empty.

  • Otherwise, if --result-file is not given, mysqltest sends test results to the standard output.

  • With --result-file but not --record, mysqltest reads the expected results from the given file and compares them with the actual results. If the results do not match, mysqltest writes a .reject file in the same directory as the result file and exits with an error. It will also print out a diff of the expected and actual result, provided it can find an appropriate diff tool to use.

  • With both --result-file and --record, mysqltest updates the given file by writing the actual test results to it. The file does not need to pre-exist.

mysqltest itself knows nothing of the t and r directories under the mysql-test directory. The use of files in those directories is a convention that is used by mysql-test-run.pl, which invokes mysqltest with the appropriate options for each test case to tell mysqltest where to read input and write output.