At MIND Research Institute, we have focused intently on studying the real-world, at-scale effectiveness of ST Math, our visual math software. In the process, we’ve gained a perspective unique among publishers on what constitutes valid effectiveness research.
That’s why we offer these guidelines on how to assess research about K-12 curricular programs.
Some of the results need to be at substantial scale, meaning at least five school-wide (not classroom) implementations. Super-motivated teachers can and will get outperforming results using ANY program. With large-scale studies (e.g. MIND has several from 25-50 schools) there are a substantial number of teachers involved, and it’s unlikely they are all champions.
Only having one year of results is not compelling. It’s very important to see whether improvement is maintained or even grows in Year Two, and to see that two different sets of students achieved similar results.
Just having outcomes improve with program use means nothing; the improvement must be greater than a similar situation without the program. Also, the added improvement from the program must withstand a test of statistical significance.
Look for large-scale results across multiple states. This means several different assessments show positive outcomes—an indication that the program is affecting something deeper than test prep for a specific type of assessment.
For a program to “stick,” you should look for gains in standardized assessments or other aspects of an accepted accountability system. Also, the research should disaggregate improvements by initial proficiency levels, so you’d know whether there was improvement among both low and high-performers.
The study should identify conditions of implementation that led to results, and you should evaluate whether these conditions—training, time, acceptance by teachers, and cost—are replicable and sustainable for you.
This post first appeared on District Administration, March 2011.
Andrew R. Coulson, Chief Data Science Officer at MIND Research Institute, chairs MIND Education's Science Research Advisory Board and drives and manages all research at MIND enterprises. This includes all student outcomes evaluations, usage evaluations, research datasets, and research partnering with grants-makers, NGOs, and universities. With a background in high-tech manufacturing engineering, he brings expertise in process engineering, product reliability, quality assurance, and technology transfer to edtech. Coulson holds a bachelor's and master's degree in physics from UCLA.
.png)
Comment