At Grasple, we are firm believers of evidence-based practices. We also recognise that gathering reliable, bias-free evidence to base decisions on is very hard, especially in education.
For solid conclusions on whether an intervention is successful or not, test-control with random selection designs are required. However, getting the (ethical) approval for test-control settings can be difficult to obtain in education.
Therefore, a lot of the research designs in education are not test-control with random selection. The chosen research design has implications for the type of questions that can be answered.
In case you do not work with a test/control design, the data can be used for:
Describing the (level of) activity of students
Exploratory (correlational) studies that yield new hypotheses and research questions
Analysing (quantitative estimates for the) quality of questions
In case you do not work with a test/control design, the data cannot be used to answer:
What the effect size of an intervention is: if there is not a control group to compare to, then it is not possible to determine what the effect of the intervention is.
What caused the effect: if there is a group to compare to (for example: data from prior years), but that group is not randomly-selected, then it is not possible to determine what caused a certain effect.
A common research design is a longitudinal study without randomly selected test/control (for example: compare to prior years without intervention to estimate the effect). This design can be used to determine the difference between years, and using that, estimate the effect size of the difference between years.
However, the design cannot be used to draw conclusions about what caused that difference.
As a teacher or researcher, you are likely very aware of these limitations.
The reason that we emphasise these limitations, is that we know how tempting it can be to try to answer important questions while those questions cannot be answered with the data available.
If you want to set up a test/control design to answer (falsify) specific hypotheses, please reach out to us at research [at] grasple [dot] com.
We are more than happy to think how we can assist in creating randomly selected test and control groups and what data we may be able to collect in Grasple for your research question.