The mysql-test-run.pl Perl script is the main application used to run the MySQL test suite. It invokes mysqltest to run individual test cases.
Invoke mysql-test-run.pl in the
mysql-test
directory like this:
shell> mysql-test-run.pl [options
] [test_name
] ...
Each test_name
argument names a test
case. The test case file that corresponds to the test name is
t/
.
test_name
.test
For each test_name
argument,
mysql-test-run.pl runs the named test case.
With no test_name
arguments,
mysql-test-run.pl runs all
.test
files in the t
subdirectory.
If no suffix is given for the test name, a suffix of
.test
is assumed. Any leading path name is
ignored. These commands are equivalent:
shell>mysql-test-run.pl mytest
shell>mysql-test-run.pl mytest.test
shell>mysql-test-run.pl t/mytest.test
A suite name can be given as part of the test name. That is, the syntax for naming a test is:
[suite_name
.]test_name
[.suffix
]
If a suite name is given, mysql-test-run.pl
looks in that suite for the test. The test file corresponding to
a test named suite_name.test_name
is
found in
suite/
.
There is also an implicit suite name suite_name
/t/test_name
.testmain
for
the tests in the top t
directory. With no
suite name, mysql-test-run.pl looks in the
default list of suites for a match and runs the test in any
suites where it finds the test. Suppose that the default suite
list is main
, binlog
,
rpl
, and that a test
mytest.test
exists in the
main
and rpl
suites. With
an argument of mytest
or
mytest.test
,
mysql-test-run.pl will run
mytest.test
from the
main
and rpl
suites.
To run a family of test cases for which the names share a common
prefix, use the
--do-test=
option. For example, prefix
--do-test=rpl
runs the
replication tests (test cases that have names beginning with
rpl
). --skip-test
has the
opposite effect of skipping test cases for which the names share
a common prefix.
The argument for the --do-test
and
--skip-test
options also allows more flexible
specification of which tests to perform or skip. If the argument
contains a pattern metacharacter other than a lone period, it is
interpreted as a Perl regular expression and applies to test
names that match the pattern. If the argument contains a lone
period or does not contain any pattern metacharacters, it is
interpreted the same way as previously and matches test names
that begin with the argument value. For example,
--do-test=testa
matches tests that begin with
testa
, --do-test=main.testa
matches tests in the main
test suite that
begin with testa
, and
--do-test=main.*testa
matches test names that
contain main
followed by
testa
with anything in between. In the latter
case, the pattern match is not anchored to the beginning of the
test name, so it also matches names such as
xmainytesta
.
To perform setup prior to running tests,
mysql-test-run.pl needs to invoke
mysqld with the --bootstrap
and --skip-grant-tables
options. If MySQL was
configured with the --disable-grant-options
option, --bootstrap
,
--skip-grant-tables
, and
--init-file
will be disabled. To handle this,
set the MYSQLD_BOOTSTRAP
environment variable
to the full path name of a server that has all options enabled.
mysql-test-run.pl will use that server to
perform setup; it is not used to run the tests.
The init_file
test will fail if
--init-file
is disabled. This is an expected
failure that can be handled as follows:
shell>export MYSQLD_BOOTSTRAP
shell>MYSQLD_BOOTSTRAP=/full/path/to/mysqld
shell>make test force="--skip-test=init_file"
To run mysql-test-run.pl on Windows, you'll
need either Cygwin or ActiveState Perl to run it. You may also
need to install the modules required by the script. To run the
test script, change location into the
mysql-test
directory, set the
MTR_VS_CONFIG
environment variable to the
configuration you selected earlier (or use the
--vs-config
option), and invoke
mysql-test-run.pl. For example (using Cygwin
and the bash shell):
shell>cd mysql-test
shell>export MTR_VS_CONFIG=debug
shell>./mysqltest-run.pl --force --timer
shell>./mysqltest-run.pl --force --timer --ps-protocol
mysql-test-run.pl uses several environment variables. Some of them are listed in the following table. Some of these are set from the outside and used by mysql-test-run.pl, others are set by mysql-test-run.pl instead, and may be referred to in tests.
Variable | Meaning |
MTR_VERSION |
If set to 1, will run the older version 1 of mysql-test-run.pl. This will affect what functionailty is available and what command line options are supported. |
MTR_MEM |
If set to anything, will run tests with files in "memory" using tmpfs or
ramdisk. Not available on Windows. Same as
--mem option |
MTR_PARALLEL |
If set, defines number of parallel threads executing tests. Same as
--parallel option |
MTR_BUILD_THREAD |
If set, defines which port number range is used for the server |
MTR_PORT_BASE |
If set, defines which port number range is used for the server |
MTR_ |
Setting of a timeout in minutes or seconds, corresponding to command
line option
-- .
Avaliable timeout names are TESTCASE ,
SUITE (both in minutes) and
START , SHUTDOWN
(both in seconds). These variables are supported from
MySQL 5.1.44. |
MYSQL_TEST |
Path name to mysqltest binary |
MYSQLD_BOOTSTRAP |
Full path name to mysqld that has all options enabled |
MYSQLTEST_VARDIR |
Path name to the var directory that is used for
logs, temporary files, and so forth |
MYSQL_TEST_DIR |
Full path to the mysql-test directory where tests
are being run from |
MYSQL_TMP_DIR |
Path to temp directory used for temporary files during tests |
The variable MTR_PORT_BASE
was added in MySQL
5.1.45 as a more logical replacement for
MTR_BUILD_THREAD
. It gives the actual port
number directly (will be rounded down to a multiple of 10). If
you use MTR_BUILD_THREAD
, the port number is
found by multiplying this by 10 and adding 10000.
Tests sometimes rely on certain environment variables being
defined. For example, certain tests assume that
MYSQL_TEST
is defined so that
mysqltest can invoke itself with
exec $MYSQL_TEST
.
Other tests may refer to the last three variables listed in the
preceeding table, to locate files to read or write. For example,
tests that need to create files will typically put them in
$MYSQL_TMP_DIR/
.
file_name
If you are running mysql-test-run.pl version
1 by setting MTR_VERSION
, note that this only
affects the test driver, not the test client (and its language)
or the tests themselves.
A few tests might not run with version 1 because they depend on
some feature of version 2. You may have those tests skipped by
adding the test name to the file
lib/v1/incompatible.tests
. This feature is
available from MySQL 5.1.40.
mysql-test-run.pl supports the options in the
following list. An argument of --
tells
mysql-test-run.pl not to process any
following arguments as options.
Display a help message and exit.
Allow tests marked as "big" to run. Tests can be thus marked
by including the line --source
include/big_test.inc, and they will only be run if
this option is given, or if the environment variable
BIG_TEST
is set to 1.
This is typically done for tests that take very long to run, or that use very much resources, so that they are not suitable for running as part of a normal test suite run.
Specify a number to calculate port numbers from. The formula
is 10 * build_thread
+ 10000.
Instead of a number, it can be set to
auto
, which is also the default value, in
which case mysql-test-run.pl will
allocate a number unique to this host.
The value (number or auto
) can also be
set with the MTR_BUILD_THREAD
environment
variable.
From MySQL 5.1.45, the more logical
--port-base
is supported as an alternative.
Instructs valgrind to use callgrind.
Check test cases for side effects. This is done by checking system state before and after each test case; if there is any difference, a warning to that effect will be written, but the test case will not be marked as failed because of it. This check is enabled by default.
The path to the directory where client binaries are located.
Start mysqltest in the ddd debugger.
Start mysqltest in the named debugger.
Start mysqltest in the gdb debugger.
The path to the directory where client libraries are located.
Extra option to pass to mysqld. The value
should consist of a single mysqld option
including dashes. This option is similar to
--mysqld
but has a different effect.
mysql-test-run.pl executes multiple test
runs, using the options for each instance of
--combination
in successive runs. If
--combination
is given only once, it has no
effect. For test runs specific to a given test suite, an
alternative to the use of --combination
is
to create a combinations
file in the
suite directory. The file should contain a section of
options for each test run. See
Section 4.9, “Passing Options from mysql-test-run.pl to
mysqld or mysqltest”.
Write str
to the output within
lines filled with #
, as a form of banner.
Compress all information sent between the client and the server if both support compression.
Pass the --cursor-protocol
option to
mysqltest (implies
--ps-protocol
).
Start mysqld in the ddd debugger.
Dump trace output for all clients and servers.
Start mysqld using the named debugger.
Controls whether the Debug Sync facility for testing and debugging is enabled. The option value is a timeout in seconds. The default value is 300. A value of 0 disables Debug Sync. The value of this option also becomes the default timeout for individual synchronization points.
mysql-test-run.pl passes
--loose-debug-sync-timeout=
to mysqld. The N
--loose
prefix is used so that mysqld does not
fail if Debug Sync is not compiled in.
For information about using the Debug Sync facility for testing, see Section 4.14, “Thread Synchronization in Test Cases”.
This option was added in MySQL 5.1.41/5.5.0/6.0.6.
Use the named file as fixed config file template for all tests.
--defaults_extra_file=
file_name
Add setting from the named file to all generated configs.
Run all test cases having a name that begins with the given
prefix
value. This option
provides a convenient way to run a family of similarly named
tests.
The argument for the --do-test
option also
allows more flexible specification of which tests to
perform. If the argument contains a pattern metacharacter
other than a lone period, it is interpreted as a Perl
regular expression and applies to test names that match the
pattern. If the argument contains a lone period or does not
contain any pattern metacharacters, it is interpreted the
same way as previously and matches test names that begin
with the argument value. For example,
--do-test=testa
matches tests that begin
with testa
,
--do-test=main.testa
matches tests in the
main
test suite that begin with
testa
, and
--do-test=main.*testa
matches test names
that contain main
followed by
testa
with anything in between. In the
latter case, the pattern match is not anchored to the
beginning of the test name, so it also matches names such as
xmainytestz
.
Use a version of mysqltest built with the embedded server.
Ignore any disabled.def
file, and run
also tests marked as disbaled. Success or failure of those
tests will be reported the same way as other tests.
Specify a file that contains a list of test cases that
should be displayed with the [ exp-fail ]
code rather than [ fail ]
if they fail.
This option was added in MySQL 5.1.33.
For an example of a file that might be specified via this
option, see
mysql-test/collections/default.experimental
.
Use an already running server. The option/value pair is what
is needed by the mysql client to connect
to the server. Each --extern
can only take
one option/value pair as argument, so it you need more you
need to repeat --extern
for each of them.
Example:
./mysql-test-run.pl --extern socket=var/tmp/mysqld.1.sock alias
Note: If a test case has an .opt
file
that requires the server to be restarted with specific
options, the file will not be used. The test case likely
will fail as a result.
Do not perform controlled shutdown when servers need to be
restarted or at the end of the test run. This is equivalent
to using --shutdown-timeout=0
.
Normally, mysql-test-run.pl exits if a
test case fails. --force
causes execution
to continue regardless of test case failure.
Run tests with the gcov test coverage tool.
Start mysqld in the gdb debugger.
Run tests with the gprof profiling tool.
--gprof
was added in 5.1.45.
Use a server that has already been started by the user in the ddd debugger.
Use a server that has already been started by the user in a debugger.
Use a server that has already been started by the user in the gdb debugger.
Marks progress with timing (in milliseconds) and line number
in
var/log/
.
testname
.progress
The maximum number of simultaneous server connections that may be used per test. If not set, the maximum is 128. Minimum allowed limit is 8, maximum is 5120. Corresponds to the same option for mysqltest.
This option is available from MySQL 5.1.45.
Limit the number of core files saved, to avoid filling up
disks in case of a frequently crashing server. Defaults to
5, set to 0 for no limit. May also be set with the
environment variable MTR_MAX_SAVE_CORE
Limit the number of data directories saved after failed
tests, to avoid filling up disks in case of frequent
failures. Defaults to 20, set to 0 for no limit. May also be
set with the environment variable
MTR_MAX_SAVE_DATADIR
Stop execution after the specified number of tests have
failed, to avoid using up resources (and time) in case of
massive failures. retries are noe counted, nor are failures
of tests marked experimental. Defaults to 10, set to 0 for
no limit. May also be set with the environment variable
MTR_MAX_TEST_FAIL
This option is not supported on Windows.
Run the test suite in memory, using tmpfs or ramdisk. This
can decrease test times significantly, in particular if you
would otherwise be running over a remote file system.
mysql-test-run.pl attempts to find a
suitable location using a built-in list of standard
locations for tmpfs and puts the var
directory there. This option also affects placement of
temporary files, which are created in
var/tmp
.
The default list includes /dev/shm
. You
can also enable this option by setting the environment
variable
MTR_MEM[=
.
If dir_name
]dir_name
is given, it is added
to the beginning of the list of locations to search, so it
takes precedence over any built-in locations.
Once you have run tests with --mem
within a
mysql-test
directory, a soflink
var
will have been set up to the
temporary directory, and this will be re-used the next time,
until the soflink is deleted. Thus, you do not have to
repeat the --mem
option next time.
Extra option to pass to mysqld. Only one
option may be specified in value
;
to specify more than one, use additional
--mysqld
options. See
Section 4.9, “Passing Options from mysql-test-run.pl to
mysqld or mysqltest”.
Pass
--ndb-connectstring=
to the master MySQL server. This option also prevents
mysql-test-run.pl from starting a
cluster. It is assumed that there is already a cluster
running to which the server can connect with the given
connectstring.
str
Disable the check for test case side effects; see
--check-testcases
for a description.
Do not reorder tests to reduce number of restarts, but run them in exactly the order given. If a whole suite is to be run, the tests are run in alphabetical order, though similiar combinations will be grouped together. If more than one suite is listed, the tests are run one suite at a time, in the order listed.
Cause mysqltest not to generate a timing file. The effect of this is that the report from each test case does not include the timing in milliseconds as it normally does.
Do not look for and report errors and warning in the server logs.
Run tests using N
parallel
threads. By default, 1 thread is used. Use
--parallel=auto
for auto-setting of
N
. The auto value was added in
MySQL 5.1.36.
Specify base of port numbers to be used; a block of 10 will
be allocated. P
should be
divisible by 10; if it is not, it will be rounded down. If
running with more than one parallel test thread, thread 2
will use the next block of 10 and so on.
If the port number is given as auto
,
which is also the default,
mysql-test-run.plwill allocate a number
unique to this host. The value may also be given with the
environment variable MTR_PORT_BASE
.
--port-base
was added in MySQL 5.1.45 as a
more logical alternative to --build-thread
.
If both are used, --port-base
takes
presedence.
Do not run any tests, but print details about all tests, in the order they would have been run.
Pass the --ps-protocol
option to
mysqltest.
Pass the --record
option to
mysqltest. This option requires a
specific test case to be named on the command line.
Reorder tests to minimize the number of server restarts needed. This is the default behavior. There is no guarantee that a particular set of tests will always end up in the same order.
Run each test N
number of times.
Display the output of SHOW ENGINES
and
SHOW VARIABLES
. This can be used to
verify that binaries are built with all required features.
If a test fails, it is retried up to a maximum of
N
runs, but will terminate after
2 failures. Default is 3, set to 1 or 0 for no retries. This
option has no effect unless --force
is also
used; without it, test execution will terminate after the
first failure.
The --retry
and
--retry-failure
options do not affect how
many times a test repeated with --repeat
may fail in total, as each repetition is considered a new
test case, which may in turn be retried if it fails.
Allow a failed and retried test to fail more than the default 2 times before giving it up. Setting it to 0 or 1 effectively turns off retries
Max number of seconds to wait for servers to do controlled shutdown before killing them. Default is 10.
Do not apply combinations; ignore combinations file or option.
Do not start NDB Cluster; skip Cluster test cases.
--skip-ndbcluster-slave
,
--skip-ndb-slave
Do not start an NDB Cluster slave.
Skip replication test cases.
Do not start mysqld with support for SSL connections.
Specify a regular expression to be applied to test case names. Cases with names that match the expression are skipped. tests to skip.
The argument for the --skip-test
option
allows more flexible specification of which tests to skip.
If the argument contains a pattern metacharacter other than
a lone period, it is interpreted as a Perl regular
expression and applies to test names that match the pattern.
See the description of the --do-test
option
for details.
--skip-*
--skip-*
options not otherwise recognized
by mysql-test-run.pl are passed to the
master server.
Pass --sleep=
to mysqltest.
N
Pass the --sp-protocol
option to
mysqltest.
If mysql-test-run.pl is started with the
--ssl
option, it sets up a secure conection
for all test cases. In this case, if
mysqld does not support SSL,
mysql-test-run.pl exits with an error
message: Couldn't find support for SSL
Initialize and start servers with the startup settings for the specified test case. You can use this option to start a server to which you can connect later. For example, after building a source distribution you can start a server and connect to it with the mysql client like this:
shell>cd mysql-test
shell>./mysql-test-run.pl --start alias &
shell>../mysql -S ./var/tmp/master.sock -h localhost -u root
If no tests are named on the command line, the server(s)
will be started with settings for the first test that would
have been run without the --start
option.
mysql-test-run.pl will stop once the server has been started, but will terminate if the server dies. If killed, it will also shut down the server.
This is similar to --start
, but will skip
the database initialization phase and assume that database
files are already available. Usually this means you must
have run another test first.
mysql-test-run.pl sorts the list of names
of the test cases to be run, and then begins with
test_name
.
Create strace output for mysqltest.
Run the named test suite. The default name is
main
(the regular test suite located in
the mysql-test
directory).
Specify the maximum test suite runtime.
Specify the maximum test case runtime.
Adds to each test report for a test case, the total time in
sconds and milliseconds passed since the preceding test
ended. This option can only be used together with
--timestamp
, and has no effect without it.
Cause mysqltest to generate a timing
file. The default file is named
./var/log/timer
.
Prints a timestamp before the test case name in each test report line, showing when the test ended.
The directory where temporary file are stored. The default
location is ./var/tmp
. The environment
variable MYSQL_TMP_DIR
will be set to the
path for this directory, whether it has the default value or
has been set explicitly. This may be referred to in tests.
The MySQL user name to use when connecting to the server.
Run mysqltest and
mysqld with valgrind.
Thiks and the following --valgrind
options
require that the executables have been build with
valgrind support.
Run the mysqld server with valgrind.
Run mysqltest with valgrind.
Extra options to pass to valgrind.
Specify the path name to the valgrind executable.
Specify the path where files generated during the test run
are stored. The default location is
./var
. The environment variable
MYSQLTEST_VARDIR
will be set to the path
for this directory, whether it has the default value or has
been set explicitly. This may be referred to in tests.
Give more verbose output regarding test execution. Use the option twice to get even more output. Note that the output generated within each test case is not affected.
Write when and why servers are restarted between test cases.
Pass the --view-protocol
option to
mysqltest.
Specify the configuration used to build MySQL (for example,
--vs-config=debug
--vs-config=release
). This option is for
Windows only.
If --start
or
--start-dirty
is used, wait for all servers
to exit before termination. Otherise, it will terminate if
one (of several) servers is restarted.
This option was added in MySQL 5.1.36.
Search the server log for errors or warning after each test
and report any suspicious ones; if any are found, the test
will be marked as failed. This is the default behavior, it
may be turned off with --nowarnings
.
Run only test cases that have ndb
in
their name.