Book review: The Trials of Evidence-based Education

Posted on 23 November 2017

The Trials of Evidence-based Education: The Promises, Opportunities and Problems of Trials in Education, by Stephen Gorard, Beng Huat See and Nadia Siddiqui

Jonathan Haslam, Deputy Director, IEE

Each year, the President of the United States gives The State of the Union address to the United States Congress. It reports on the state of the nation and outlines the President’s national priorities and plans for the future.

The Trials of Evidence-based Education aims to take a similar approach on the progress of education research. To make it an annual event might be over-ambitious, but a repeat performance in five years would be fascinating.

The authors are critical of much education research – “such poor quality that it might do more harm than good” – “a majority of published and unpublished ‘research’ is of no consequence or use for any real-life purpose”. Thankfully, they provide some useful pointers for spotting the good stuff.

They are very critical of the use of significance tests (ie, p=<0.05), which purport to show the likelihood that a result is not due to chance. These tests, they say, “just do not work”, and propose instead using the “number of counterfactual cases needed to disturb the finding (NNTD)” as a measure of sensitivity. The resolution of this debate within the research community seems unlikely to happen any time soon, so for those trying to read and appraise research, for now, this highlights the importance of looking at other aspects of a study to gauge reliability. The book provides a useful “sieve” for estimating the trustworthiness of a research study (see below). Any study is only as good as its lowest rating.

The second half of the book looks at the results of a series of studies the authors have conducted. Each looks at what we know from previous research on the approach, followed by the findings from the current study. The results are then combined in a handy table, and the implications for practice are drawn out.

  • Catch-up interventions – Switch-on Reading, Accelerated Reader and Fresh Start look like good options; commercial software alone should be avoided; Response to Intervention deserves another chance; and summer schools might be worth it for other reasons, but not for improving literacy.
  • Whole school approaches to improving learning – Surprisingly (perhaps not!), given the current policy trend, learning to reason beats learning facts. Philosophy for Children is worth consideration; a wider liberal curriculum (Core Knowledge) to teach literacy does not work; enhancing feedback alone is also not promising, although the weight of prior evidence suggests it has some value.
  • Enrichment activities – the impact of Youth United, the Children’s University or Philosophy for Children on non-cognitive outcomes is not strong, with Philosophy for Children just shading it.

The final section of the book summarises some conclusions that can be drawn, or perhaps challenges to current thinking. In particular, they observe that the successful interventions take control away from the teacher, either by giving it to the child (Philosophy for Children) or by providing strict protocols for teachers to adhere to (Switch-on Reading). Improving less successful teachers has not been shown to work, or is at least proving to be more difficult, as is encouraging teachers to engage with research.

A more familiar challenge is that everyone is almost always convinced that an intervention works – developers, teachers, parents and children. “People involved just cannot tell whether something works, and therefore simply asking them is no kind of evaluation at all.” The book is full of gems like this. That can make the book hard reading, because it makes you think, which, I hope, means I learned something, although now I’m not quite sure.

If I have a criticism of the book, it is that the references are not always clear (in order to find out more about a particular trial, for example) but that is a quibble. I look forward to the next State of the Union in a few years’ time.

For a different perspective on this book, have a look at the review by Stephen Foreman, Research-lead at Huntington Research School.

Leave a reply

Your email address will not be published.