At MIND Research Institute, we have focused intently on studying the real-world, at-scale effectiveness of ST Math, our visual math software. In the process, we’ve gained a perspective unique among publishers on what constitutes valid effectiveness research.
That’s why we offer these guidelines on how to assess research about K-12 curricular programs.
Some of the results need to be at substantial scale, meaning at least five school-wide (not classroom) implementations. Super-motivated teachers can and will get outperforming results using ANY program. With large-scale studies (e.g. MIND has several from 25-50 schools) there are a substantial number of teachers involved, and it’s unlikely they are all champions.
Only having one year of results is not compelling. It’s very important to see whether improvement is maintained or even grows in Year Two, and to see that two different sets of students achieved similar results.
Just having outcomes improve with program use means nothing; the improvement must be greater than a similar situation without the program. Also, the added improvement from the program must withstand a test of statistical significance.
Look for large-scale results across multiple states. This means several different assessments show positive outcomes—an indication that the program is affecting something deeper than test prep for a specific type of assessment.
For a program to “stick,” you should look for gains in standardized assessments or other aspects of an accepted accountability system. Also, the research should disaggregate improvements by initial proficiency levels, so you’d know whether there was improvement among both low and high-performers.
The study should identify conditions of implementation that led to results, and you should evaluate whether these conditions—training, time, acceptance by teachers, and cost—are replicable and sustainable for you.
This post first appeared on District Administration, March 2011.
Andrew R. Coulson is Chief Data Science Officer at MIND Research Institute. His team of data analysts evaluate program usage and measure student learning outcomes. Follow Andrew on Twitter at @AndrewRCoulson.
.png)
Comment