Does That Program Really Work? 6 Questions to Ask About Research

At MIND Research Institute, we have focused intently on studying the real-world, at-scale effectiveness of ST Math, our visual math software. In the process, we’ve gained a perspective unique among publishers on what constitutes valid effectiveness research.  

That’s why we offer these guidelines on how to assess research about K-12 curricular programs. 


1. What was the size of the study?

Some of the results need to be at substantial scale, meaning at least five school-wide (not classroom) implementations. Super-motivated teachers can and will get outperforming results using ANY program. With large-scale studies (e.g. MIND has several from 25-50 schools) there are a substantial number of teachers involved, and it’s unlikely they are all champions.


2. Are the results repeatable over years?

Only having one year of results is not compelling. It’s very important to see whether improvement is maintained or even grows in Year Two, and to see that two different sets of students achieved similar results. 


3. What were the results compared to?

Just having outcomes improve with program use means nothing; the improvement must be greater than a similar situation without the program. Also, the added improvement from the program must withstand a test of statistical significance.


4. Are the results repeatable across states?

Look for large-scale results across multiple states. This means several different assessments show positive outcomes—an indication that the program is affecting something deeper than test prep for a specific type of assessment.


5. What outcomes were measured, and for which students?

For a program to “stick,” you should look for gains in standardized assessments or other aspects of an accepted accountability system. Also, the research should disaggregate improvements by initial proficiency levels, so you’d know whether there was improvement among both low and high-performers.


6. Is the program implementation replicable and sustainable?

The study should identify conditions of implementation that led to results, and you should evaluate whether these conditions—training, time, acceptance by teachers, and cost—are replicable and sustainable for you.


This post first appeared on District Administration, March 2011.

Andrew Coulson

About the Author

Andrew R. Coulson, Chief Data Science Officer at MIND Education, specializes in digital data collection techniques and conducting rigorous evaluations of organic quasi-experiments. With a background in high-tech manufacturing engineering for over 17 years, he brings expertise in operations, process engineering, product reliability, and technology transfer. Coulson holds a bachelor’s and master’s degree in physics from UCLA.


Join Our Newsletter