For too long, we’ve held just teachers and “teacher quality,” accountable for student outcomes. The glare of the student results spotlight has been intensely focused on teachers alone despite compelling evidence that instructional content and programs is almost as important to student learning. (In a scathing white paper, “Choosing Blindly, Instructional Materials, Teacher Effectiveness, and the Common Core,” Matthew Chingos and Grover Whitehurst blast programs and lack of training and development.)
Not only is it past time to hold content and programs accountable, but we are out of excuses not to. A recent study published by WestEd demonstrates that cost-effective and rigorous evaluations of new programs can now be pursued at any time in any state.
The diversity of content – scope, vehicles, approaches and instructional design – available today is far greater than when teacher-selection-committees simply chose among “Big 3” publishers’ textbooks. Spearheaded by higher standards, as a nation we are aiming to transform student learning outcomes to be much deeper than in the past. Yet, the only things which are going to change are content and programs: the power of the tools and training we provide to our teachers.
Technology-driven revolutions happen when people readily adopt new, immensely more powerful tools to get work they wanted to get done, done. This came naturally for printed books, spreadsheets, email and smart phones. But in education, it’s been extremely challenging to determine what tools actually work, in what contexts and to what ends. This is due to gigantic variability in tool use and school culture and has led, understandably, to skepticism about replicating anecdotal results. Instead we need credible evaluations, diving as deep as randomized student-level. But these are complex, logistically challenging, high cost, and notoriously sparse and slow.
Now that states all have testing systems in grades 3-8, there is consistent grade-level information about proficiency rates and some ability to measure growth rates, enabling any content or program to show its ability to add value, shortly after state assessment results are released each year.
This grade-level evaluation method is straightforward and replicable across years, states and program-types. It also works for every user (school site) in a state, taking into account all real-world variability, easily reporting out on hundreds of schools and tens of thousands of students.
To use the method, the program must be:
When these conditions are met, a study that meets What Works Clearinghouse standards of rigor are possible without prior planning, as this WestEd study of ST Math results shows. Every program, in every state, every year, that meets the above criteria can be studied, whether for the first time or for replication. The data is waiting in state and NCES research files to be used, in conjunction with complete and accurate records of program usage.
It may be too early for this level of accountability to be palatable for many publishers just yet. Showing robust, positive results will require true program efficacy. There will be many findings of small effect sizes, many implementations which fail, and much lack of statistical significance. Third-party factors may confuse the results. Publishers would need to report out failures as well as successes. But the alternative is to continue to rely on peer word-of-mouth recommendations.
When this research method becomes the industry norm, imagine the renewed competition to improve the tools we give our teachers. We must hold ourselves responsible for imposing accountability in education. Only then will we have a real education revolution.
This opinion piece was originally published on EdWeek.
Andrew R. Coulson, Chief Data Science Officer at MIND Research Institute, chairs MIND Education's Science Research Advisory Board and drives and manages all research at MIND enterprises. This includes all student outcomes evaluations, usage evaluations, research datasets, and research partnering with grants-makers, NGOs, and universities. With a background in high-tech manufacturing engineering, he brings expertise in process engineering, product reliability, quality assurance, and technology transfer to edtech. Coulson holds a bachelor's and master's degree in physics from UCLA.
.png)
Comment