When unitize is run with a test file against an existing unitizer store, each test in the file is matched and compared to the corresponding test in the store. Here is a comprehensive list of possible outcomes:
unitizer store no longer exists in the test file so you will be prompted to remove it from the storeunitizer_sect for more details on custom comparison functions). Because the comparison function itself failed, unitizer has no way of knowing whether the test passed or failed; you can think of it as an NA outcome.When reviewing tests, unitizer will group tests by test type, so you will review all new tests in one go, then the failed tests, and so on. As a result, the order that you review tests may not be the same as the order they appear in in the test file.
As noted previously simple assignments are not considered tests. They are stored in the unitizer store, but you are not asked to review them, and their values are not compared to existing reference values prior to storage. The implicit assumption is that if there is an assignment the intent is to use the resulting object in some later test at which point any issues will crop up. Skipping assignment review saves some unnecessary user interaction.
You can force assignments to become tests by wrapping them in parentheses:
a <- my_fun(25) # this is not a test
(a <- my_fun(42)) # this is a testThe actual rule unitizer uses to decide whether an expression is a test or not is whether it returns invisibly. Wrapping parentheses around an expression that returns invisibly makes it visible, which is why assignments in parentheses become tests. Conversely, you can wrap an expression in invisible(...) to prevent it from being treated as a test.
unitizer Test ComponentsThe following aspects of a unitizer tests are recorded for future comparison:
invokeRestart (e.g. was stop called in the expression)Currently only the first two elements are actually compared when determining whether a test passes or fails. These two should capture almost all you would care about from a unit test perspective.
Screen output is omitted from comparison because it can be caused to vary substantially by factors unrelated to source code changes (e.g. console display width). Screen output will also seem identical to the value as most of the time screen output is just the result of printing the return value of an expression. This will not be the case if the expression itself prints to stdout explicitly, or if the function returns invisibly.
Message output is omitted because all typical mechanisms for producing stderr output also produce conditions with messages embedded, so it is usually superfluous to compare them. One exception would be if an expression cated to stderr directly.
The “abort” invokeRestart is omitted because it generally is implied by the presence of an error condition and actively monitoring it clutters the diagnostic messaging produced by unitizer. It exists because it is possible to signal a “stop” condition without actually triggering the “abort” restart so in some cases it could come in handy.
While we omit the last three components from comparison, this is just default behavior. You can change this by using the compare argument for unitizer_sect.
untizer_sectOften it is useful to group tests in sections for the sake of documentation and clarity. Here is a slghtly modified version of the original demo file with sections:
unitizer_sect("Basic Tests", {
library(unitizer.fastlm)
x <- 1:10
y <- x ^ 3
res <- fastlm(x, y)
get_slope(res)
})
unitizer_sect("Advanced Tests", {
2 * get_slope(res) + get_intercept(res)
get_rsq(res)
})Now re-running unitizer segments everything by section (note, first few lines are set-up):
(.unitizer.fastlm <- copy_fastlm_to_tmpdir())
update_fastlm(.unitizer.fastlm, version="0.1.2")
install.packages(.unitizer.fastlm, repos=NULL, type='src', quiet=TRUE)
unitize(file.path(.unitizer.fastlm, "tests", "unitizer", "unitizer.fastlm.R"))
+------------------------------------------------------------------------------+
| unitizer for: tests/unitizer/unitizer.fastlm.R |
+------------------------------------------------------------------------------+
Pass Fail New
1. Basic Tests - - 1
2. Advanced Tests - - 2
..................................
- - 3
If there are tests that require reviewing, each section will be reviewed in turn.
Note that unitizer_sect does not create separate evaluation environments for each section. Any created object will be available to all lexically subsequent tests, regardless of whether they are in the same section or not.
It is possible to have nested sections, though at this point in time unitizer only explicitly reports information at the outermost section level.
By default tested components (values and conditions) are compared with all.eq, a wrapper around all.equal that returns FALSE on inequality instead of a character description of the inequality. If you want to override the function used for value comparisons it is as simple as creating a new section for the tests you want to compare differently and use the compare argument:
unitizer_sect("Accessor Functions", compare=identical,
{
get_slope(res)
get_rsq(res)
get_intercept(res)
} )The values produced by these three tests will be compared using identical instead of all.eq. If you want to modify how other components of the test are compared, then you can pass a unitizerItemTestsFuns object as the value to the compare argument instead of a function:
unitizer_sect("Accessor Functions",
compare=unitizerItemTestsFuns(
value=identical,
output=all.equal,
message=identical
),
{
get_slope(res)
get_rsq(res)
get_intercept(res)
} )This will cause the value of tests to be compared with identical, the screen output with all.equal, and messages (stderr) with identical.
If you want to change the comparison function for conditions, keep in mind that what you are comparing are conditionList objects so this is not straightforward (see getMethod("all.equal", "conditionList")). In the future we might expose a better interface for custom comparison functions for conditions (see issue #32).
If you need to have different comparison functions within a section, use nested sections. While unitizer will only report the outermost section metrics in top-level summaries, the specified comparison functions will be used for each nested section.
Whenever you re-run unitize on a file that has already been unitized, unitizer matches the expressions in that file to those stored in the corresponding unitizer store. unitizer matches only on the deparsed expression, and does not care at all where in the file the expression occurs. If multiple identical expressions exist in a file they will be matched in the order they show up.
The unitizer_sect in which a test was when it was first unitized has no bearing whatsoever on matching a new test to a reference test. For example, if a particular test was in “Section A” when it was first unitized, but in the current version of the test file it is in “Section X”, that test will be matched to the current one in “Section X”.
unitizer parses the comments in the test files and attaches them to the test that they document. Comments are attached to tests if they are on the same line as the test, or in the lines between a test and the previous test. Comments are displayed with the test expression during the interactive review mode.
In order to properly capture output, unitizer will modify streams and options. In particular, it will do the following:
options(warn=1L) during expression evaluationoptions(error=NULL) during expression evaluationsink() to capture any output to stdoutsink(type="message") to capture output to stderrThis should all be transparent to the user, unless the user is also attempting to modify these settings in the test expressions. The problematic interaction are around the options function. If the user sets options(warn=1) with the hopes that setting will persist beyond the execution of the test scripts, that will not happen. If the user sets options(error=recover) or some such in a test expression, and that expression throws an error, you will be thrown into recovery mode with no visibility of stderr or stdout, which will make for pretty challenging debugging. Similarly, unitizeing debugged functions, or interactive functions, is unlikely to work well.
If unitize is run with sdtderr or stdout sunk, then it will subvert the sink during test evaluation and reset it to the same sinks on exit. If a test expression sinks either stream, unitizer will stop capturing output from that point on until the end of the test file. At that point, it will attempt to reset the sinks to what they were when unitizer started. Sometimes this is not actually possible. If such a situation occurs, unitizer will release all sinks to try to avoid a situation where control is returned to the user with output streams still captured.
To reduce the odds of storing massive and mostly useless stdout, unitize limits how much output is stored by default. If you exceed the limit you will be warned. You may modify this setting with options("unitizer.max.capture.chars").