In each episode of the Inside Our MIND podcast, we take a look at issues and challenges facing education that we are working to address through research, technology and strategic initiatives.
In our latest episode, Chief Data Science Officer Andrew Coulson returns to the show to discuss the topic of statistical significance. Andrew provides an overview of the concept, and explains how taking a binary perspective toward statistical significance can lead to misinterpretation and misinformation. He and Brian discuss the American Statistical Association's recommendation to get rid of the term entirely, which is gaining support in the scientific community. They also talk about a recent evidence review which ignored promising study results because of an “all or nothing” view of statistical significance. As always, Andrew provides some takeaways for educators about looking beyond one metric or one study when gathering information on a tool or resource for districts, schools, and classrooms.
That treatment might very well be effective, and the experiment might measure that it had some positive effect, but the statistical test of significance says “we can't be at least 95% sure that we weren't fooled.” So, even though is does have an effect, it gets lumped into that “we're going to ignore it” pile.
-Andrew Coulson, MIND Research Institute
You can listen to the episode in the player below:
Topics Covered in the Podcast:
3:15 - What is Statistical Significance?
4:30 - The Problem with a Binary Mindset
6:15 - Misinterpretations and Missing Results
9:00 - ASA Moves Away from Statistical Significance
12:00 - Factors that Affect Statistical Significance
13:20 - Don't Rest on One Result
17:30 - Takeaways
Thanks for listening to the podcast! Please leave us a review on iTunes, Google Podcasts, Spotify, Spreaker or wherever you are listening to the show. Subscribe to get future episodes as soon as they are released!
Ebook: Demanding More from Edtech Evaluations
Video: What is Effect Size?