Education research and a load of old cobblers
Jonathan Haslam, Deputy Director, IEE
(This post is inspired by Susie Dent’s origins of words section in the TV show Countdown.)
On a recent episode of the show she recounted the history of the word ultracrepidarian. The word comes from the story of a Greek painter, Apelles, who would leave his paintings in the front of his shop and listen for the comments of passers-by. One day a shoemaker noticed that in one of the paintings a pair of sandals hadn’t been painted quite correctly. Apelles listened to his criticism and repainted the sandals. Emboldened by this success, the next day the shoemaker criticised other aspects of the painting, at which point Apelles replied (as recounted in the Latin telling of the tale) “Sutor, ne ultra crepidam” – shoemaker, not beyond the shoe. In other words, don’t comment beyond your area of expertise. To this day, ultracrepidarian means, according to the OED, “Going beyond one’s proper province; giving opinions on matters beyond one’s knowledge”. I was delighted to find that there is a word for this, but unfortunately it is a wretchedly difficult word to say, so it is easy to see why it is not commonly used.
Anyone interested in the use of evidence in education is familiar with people who give their opinions on matters beyond their knowledge. Actually, anyone interested in education is familiar with them too, but for the purposes of this post, I want to think about the appropriate use of evidence.
“When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.”
The Daily Mail headline
It is dangerous to take the results of an individual study and extrapolate them too far, particularly when the conditions of the experiment are very different from your own. For example, the Daily Mail headline “How red wine could slow down ageing” was derived from a study of the impact of a chemical called resveratrol on mice. The chemical is also found in red wine, but in concentrations so low it would be physically impossible to drink sufficient wine to have the same effect in humans, even if there were the same effect.
A recent example in education was a study that showed that students who were allowed a laptop or tablet in class performed worse than students who were not. The study was carried out on students, with an average age of 20, at the United States Military Academy (West Point). That is not to say that the results will not hold true for Year 11 students in Stoke, but there may be differences with West Point cadets.
This rough guide to spotting bad science is a handy way of distinguishing the truth from the headlines. Particularly important is the lack of replication in education research. Fads, fashions, and entire careers have been built on the findings of a limited number of studies.
Filling in the gaps
Not everything in education has been researched, but schools need an answer to everything. When research is lacking, it makes sense to use professional judgement to come up with a solution. Difficulties arise when experts (whether practitioners, academics or policymakers) then fail to distinguish between advice and guidance that is evidence-based or evidence-light. There is nothing wrong in saying “I tried this and it worked for me” provided that the implications of this small-scale experiment are not over-claimed.
The best evidence-based guidance is open about the strength of the evidence it relies upon.
- The EEF toolkit uses padlocks to show the evidence strength behind its recommendations for a particular strand. Their guidance reports include similar information.
- What Works Clearinghouse practice guides provide information on the amount of evidence supporting a particular recommendation.
- Interventions on the IEE’s Evidence 4 Impact website must show robust evidence of impact before receiving a positive rating.
So what are the implications?
When communicating about research, be honest and open about its limitations and look for this openness and honesty in others.
If an author claims that an approach is evidence-based, expect to see references to research throughout.
Don’t be an ultracrepidarian, and don’t talk a load of old cobblers.