rcrtan

Geoffrey T. LaFlair

2018-12-18

This package contains functions for criterion-referenced test and item analysis.

Test Analysis: Dependability

The functions below can be used to examine the dependability of single administration CRT assessments.

subkoviak

The subkoviak function follows Subkoviak’s (1988) single administration procedures for estimating \(\kappa\) and po. To use the subkoviak function the data can come in one of two forms:

  1. A dataframe of dichotomously scored items
  2. A dataframe of total (raw) scores on the test. This also requires the analyst/researcher to know how many (dichotomous) items were on the test.

In both situations, the analyst/researcher needs to know the raw cut score of the test.

The procedure uses two pieces of information to estimate \(\kappa\) and po. They are a reliability coefficient (estimated in this function using Kuder-Richardson 20 or 21) and a standardized cut-score, which is estimated following Subkoviak (1988). The estimated \(\alpha\) coefficient and standardized cut-score are used to look up the values for \(\kappa\) and po in look up tables, which the function will return if look_up = TRUE

\[ z = \frac{c - 0.5 - M}{S}\]

where:

Examples: With item information

The output from the analysis contains five variables.

  1. z: The actual estimated standardized cut-score
  2. z_rounded: The rounded standardized cut-score used in the table look up
  3. KR_est: The estimated alpha coefficient using one the Kuder-Richardson methods
  4. agree_coef.r_0.4: The estimated agreement (po) coefficient
  5. kappa_coef.r_0.4: The estimated kappa (\(\kappa\)) coefficient

Examples: Without item information

When item-level information is not available the items argument takes the number of dichotomous items on the test, and the total argument takes the name of the column in the dataframe with the total score.

phi_domain

The phi_domain function calculates Brown (1988) short-cut estimate of the generalizability coefficient (\(\phi\) dependability) for absolute decisions.

phi_lambda

The phi_lambda function uses Brennan (1984) formula to estimate the dependability of a test. One difference to note between this function and subkoviak is that the cut_score argument is expressed as a proportion.

References

Brennan, R. L. (1984). Estimating the dependability of the scores. In A guide to criterion-referenced test construction, (Ed.) Ronald A Berk. 231–66. Baltimore, MD: The Johns Hopkins University Press.

Brown, J. D. (1988). Short cut estimates of criterion-referenced test reliability. Language Testing, 7, 77–97.

Subkoviak, M. J. (1988). A practitioner’s guide to computation and interpretation of reliability indices for mastery tests. Journal of Educational Measurement, 25, 47–55.