If you haven’t read the general introduction to justifier yet, you may want to start there.
Planning a scientific study in psychology entails taking very many decisions. Some of these decisions are taken without thinking about it: for example, given the default to use either Type II (SPSS) or Type III (R) Sums of Squares in Analysis of Variance analyses, users of either package will sometimes follow that default without realising this constitutes an implicit decision. However, ideally, researchers make these decisions deliberately, and ideally, they have justifications for these decisions.
In fact, a recent blog post called upon researchers in psychology to #JustifyEverything.
And indeed, if researchers would justify all their decisions, this has the potential of yielding a wealth of benefits. First, researchers may become more aware of the decisions they take, enabling them to take those decisons more deliberately. This would decrease the much-lamented unquestioning reliance on defaults in the used software. Second, review of study proposals has the potential to become much more efficient, if the decisions that were taken and their justifications can be extracted and parsed automatically. Third, once justifing everything becomes common practice, analysing these decisions and their justifications can be very insightful, for example showing where training of researchers may be most beneficial (i.e. areas where decisions and their justifications are based on outdated assumptions).
However, for this potential to be realised, justifications need to be documented in a uniform format. Ideally, this is a format that is simultaneously userfriendly enough to be used by humans as they document their meeting minutes or notes, and structured enough to be machine-readable so as to enable automatic extraction and parsing. The justifier package provides this format. The basic introduction of justifier is in the general introduction vignette, and this vignette presents a simple justification framework for planning psychological studies.1
In this example framework, we will limit ourselves to four decisions that a researcher always takes for any study they undertake. The first is to establish the research question or questions, or if appropriate, hypothesis or hypothesis. The second is to determine which method wil be used. The third is deciding how many data points need to be sampled (for example, how many participants will be recruited). The fourth is whether to preregister the study.2
The example study is an applied study into the social-cognitive determinants of getting one’s ecstasy pills tested. In the Netherlands, there are drug test labs where once can bring drugs to anonymously get those tested and find out the ingredients and their dose. Ectasy pills ideally contain only one active ingredient: 3,4-methylenedioxymethamphetamin (MDMA). However, pills can contains varying doses of MDMA (in 2017, the mean MDMA dose was 187mg of MDMA; 1% of the pills had a dose under 35mg of MDMA and 70% over 140mg of mDMA). In addition, pills can contain contaminants (in 2017, 87.6% of the pills tested in the Netherlands contained only MDMA as active ingrdient).
This example is chosen for two reasons. First, it’s my topic, so this way I don’t have to learn about another domain to write it :-) Second, it’s the same example as the one used in the intervention development vignette, so that readers won’t have to familiarize themselves with two scenarios.
The first decision is in a way the simplest one: what exactly does one want to study? However, this research question isn’t arbitrary. In this world of finite, often scarce, resources, justification of why to do a study in the first place is perhaps one of the most important considerations.
In this applied example study, the research question is "What are the most important determinants of getting one's ecstasy tested?". The answer to this research question is required to develop and effective interventions to promote ecstasy pill testing. It is important to promote ecstasy pill testing because if ecstasy users do not get their pills tested, they are exposed to two risks. First, they may unknowingly ingest contaminants; and second, proper dosing (~ 1-1.5 mg of MDMA per kg of body weight) becomes hard if the dose in a pill is unknown. To minimize the likelihood of incidents and accidental intoxication, it is important that ecstasy users get their pills tested.
Often, the appropriate method follows from a combination of the type of research question and knowledge about the relevant domain. Some research questions require qualitative research; some quantitative research; and some are best answered by research synthesis. However, within each of these, many choices have to be made. To name a few examples, for qualitative research, which analysis approach is most appropriate and how to sample participants; for quantitative research, whether to use an experimental design or not and which operationalisations to use; and for a research syntesis, which bibliographic databases to use and whether to extract effect size esimates.
In the example study, there are two kinds of considerations. On the one hand, logistical considerations (available time and money. On the other hand, available prior knowledge from earlier research. On the one hand, one needs to know enough about potential determinants and sub-determinants to develop a sufficiently comprehensive questionnaire. On the other hand, one can keep doing qualitative studies and learning a little bit more ad infinitum, while in reality, one often has to work with (severely) limited funds and time.
decision:
context: global_study_method
value: quantitative
id: study_method
label: "We will conduct a quantitative study."
justification:
-
id: enough_known_about_testing_determinants
label: "We have enough information available to develop a questionnaire that is likely to measure the most important determinants and sub-determinants of XTC testing."
assertion:
-
id: testing_xtc_is_reasoned
source: phd_peters_2008
label: "Previous research indicates that getting one's ecstasy tested (or not) is largely a reasoned behavior"
-
id: there_is_qualitative_research_about_testing
source: phd_peters_2008
label: "There exists qualitative research about why people get their ecstasy tested"
-
id: limited_time
label: "We have to produce the ultimate intervention before the end of 2019."
source:
id: project_propoal
label: "The original proposal for this project as funded by our funder."Once the method is clear, for empirical research, it becomes possible to plan the required sample size. For example, in this example study, we aim to map the importance of a series of behavioral determinants to then inform intervention development. That means we desire sufficiently accurate estimates of determinants’ relevance.
decision:
context: global_study_method
value: 400
id: study_sample_size
label: "We aim to recruit around 400 participants."
justification:
-
id: enough_known_about_testing_determinants
label: "We have enough information available to develop a questionnaire that is likely to measure the most important determinants and sub-determinants of XTC testing."
assertion:
-
id: testing_xtc_is_reasoned
source: phd_peters_2008
label: "Previous research indicates that getting one's ecstasy tested (or not) is largely a reasoned behavior"
-
id: there_is_qualitative_research_about_testing
source: phd_peters_2008
label: "There exists qualitative research about why people get their ecstasy tested"
-
id: limited_time
label: "We have to produce the ultimate intervention before the end of 2019."
source:
id: project_propoal
label: "The original proposal for this project as funded by our funder."Preregistation is an excellent example of a decision that is taken for every single study, yet often without the researchers’ awareness.
justifier formatNote that this framework is intended as an example, not as an authoritative framework that should be relied on in the wild.↩
Again, this is a gross oversimplification; these are not the most important decisions, but they have been chosen because they cover different kinds of decisions, with different kinds of evidence available.↩