But educators shouldn’t rely solely on this tier system. An edtech program can be rated as having “strong” evidence based on a single randomized control trial, done many years ago, on a version of a product that no longer exists.
That study might also be on a state assessment that hasn’t been around for years, or on a small sample and a specific type of school or student body. And because these ratings do not expire, companies can stop submitting their products for evaluation once they obtain a single “gold standard” study that falls into the first tier.
Because rigorous studies have historically been rare, it’s still up to the individual to critically look for the pattern of evidence for each product they are considering.
Aggregated website lists may be a good beginning for someone starting to look for primary source information, but it is still up to the individual educator to ensure that the studies quoted are applicable enough to their school or district, recent enough, and effective based on relevant assessments.
So Where Should Educators Look?
The answer to the question is not so much as a “where to look,” but “what to look for.” Here at MIND, we believe a high volume of effectiveness studies is the future of a healthy market of product information in education.
We believe that a truly effective program will:
Share a large number of studies
Show study replicability
Use recent program versions
Show effect sizes as reflected by multiple state assessments
Cover many varied districts and school situations, subgroups and usage models
In order to know whether a program will be effective for your district, your schools and your students, you need to look for studies (multiple studies!) that reflect your population, your assessments, and the version of the program you are looking to start using.
See below for more resources on how to evaluate EdTech and our philosophy:
Andrew R. Coulson is Chief Data Science Officer at MIND Research Institute. His team of data analysts evaluate program usage and measure student learning outcomes. Follow Andrew on Twitter at @AndrewRCoulson.