Independent education research firm WestEd recently published the largest ever national study evaluating a math edtech program. The study found significant results at schools that used ST Math, a PreK-8 visual instructional program that builds a deep conceptual understanding of math through rigorous learning and creative problem solving. For schools that implemented ST Math consistently, the results were especially significant.
See an overview of the study in the following infographic:
Two outcomes were evaluated: average math scale scores and the proportion of students who were proficient or above in math. For both measures, grades that consistently implemented ST Math improved significantly more than similar grades that didn’t use the program. For example, a school that initially ranked at the 50th percentile in its state and then consistently implemented ST Math, would, on average, get a statistically adjusted boost of 14 percentile points in statewide ranking.
The effect size of all grades combined equaled 0.36, surpassing the Federal What Works Clearinghouse criteria for “substantively important” effect. This means, generally speaking, that using ST Math made a meaningful difference in schools’ math achievement.
The 14 states in the high fidelity group were California, Colorado, Connecticut, Florida, Georgia, Iowa, Michigan, Missouri, New Jersey, Nevada, New York, Texas, Virginia, Wisconsin.
Since every state has its own standardized test, comparing test scores across state lines is like comparing apples to oranges. To normalize them, the researchers z-scored each school, grade by grade, within the state – essentially providing a statewide ranking. They then looked at how each grade at the school scored within their state before and after using ST Math, and compared it to how similar schools’ scores changed within their state during the same timeframe. With all 14 states’ math tests transformed to the same basis (statewide z-score) it was possible to aggregate and compare data across any state or assessment.
Rather than selecting specific districts or a single state, this study examined all grades 3, 4 and 5 at all schools using ST Math in 2015-16 in 36 states. Schools had to have used ST Math for 1 to 3 years and state grade-average standardized test scores had to be complete and available for those years.
Schools were included in this group if at least 85% of students in the grade used ST Math, and if those students covered, on average, at least half of their ST Math grade-level content. As a general guideline, most students can complete their grade-level program if they use ST Math for about 20 minutes every weekday, or 90 minutes/week either at home or at school. The report also separately evaluated all 80,000 ST Math students regardless of implementation and found a significant effect size of 0.17.
Like other rigorous studies, WestEd carefully matched the treatment group (which used ST Math) and a similar control group (which did not use ST Math). To create the control group, WestEd statisticians matched each ST Math grade at a school to the most similar grade at another school in the same state based on size, ethnic composition, percent free and reduced lunch, and baseline year test performance. Control schools were not filtered based on math programs being used.
Read the full report from WestEd.
Andrew R. Coulson, Chief Data Science Officer at MIND Research Institute, chairs MIND Education's Science Research Advisory Board and drives and manages all research at MIND enterprises. This includes all student outcomes evaluations, usage evaluations, research datasets, and research partnering with grants-makers, NGOs, and universities. With a background in high-tech manufacturing engineering, he brings expertise in process engineering, product reliability, quality assurance, and technology transfer to edtech. Coulson holds a bachelor's and master's degree in physics from UCLA.
.png)
Comment