“It’s this big volume of information that they have to cram into their heads which is a big source of the stress.” Dr. Denise Burns quoted in The Irish Examiner
Today is Leaving Cert results day, and around 57,000 candidates – almost all of them school-leavers- will be opening that long-anticipated envelope with either joy or tears or something in between. The media will be out in force covering this event and there will be words from Minister Richard Bruton, whose department is currently working on changes to senior cycle curriculum and assessment.
In a prelude to today’s results, the Leaving Cert was also in the news last Monday when Dr. Denise Burns of DCU was interviewed on air and in the newspapers about a study which she led and which will be available to the general public next month. You can read it here, but it is behind a paywall. You might ask why the story featured so heavily on Monday rather than next month when interested listeners or readers could access the information more easily, but in Ireland education tends to be a seasonal rather than a year-round news item.
The study has two parts, one that looked at the examination papers themselves and another that interviewed recent candidates, and this blog also has two parts: the study itself and the media coverage.I should say that in relation to the study itself, these are my own thoughts and initial reactions rather than a detailed critique of its methodology.
There is a degree of disparity between the study as published and the results as reported in the media, both by journalists and by the authors themselves in direct interviews. The most striking of these is the referral to “grind-schools” in all the articles and interviews even though the report itself does not mention grind-schools or the word “grind”. Articles and interviews also fail to distinguish between full-time “grind-schools” (which students attend instead of more conventional voluntary or ETB schools) and supplementary grinds in individual subjects. There is also a sense that this study proves that all anyone does in preparation for the Leaving Cert is to learn off paragraph-shaped chunks of notes and even whole essays (“students are learning off essays, textbooks and notes” Irish Examiner) but it actually states:
“While the anecdotal reports would strongly suggest that preparation for the examination is dominated by a reliance on reproducing text verbatim, the evidence base for this assertion is limited.”
The analysis of the examination papers consisted of looking at “command verbs” in questions and inferring from these what kind of knowledge the question was assessing, and whether the skills involved in answering were “higher order skills”.
“In an in-depth document analysis of the examination papers, 14, 910 occurrences of command verbs were coded for the intellectual skill and knowledge domains required by the assessment task.” (abstract)
Now while to my uninitiated eyes this part of the study is rigorously designed and looked at a large number of exam papers, I am not sure it forms a secure basis for the claims being made in the newspapers. (I know over-simplification and hyperbole are factors common to all media reporting of all academic research as anyone who’s read Ben Goldacre’s excellent work in this area can testify). It rests on definitions and taxonomies of knowledge and skills that are “widely accepted” but not uncontested. An underlying premise is that “studies suggest that the general intellectual skills are essentially the same across disciplines” and the study deals with examination papers across the humanities, the sciences, maths, home economics and the arts treating them all to the same analysis. The only significant omission is languages other than English. This wide-range is laudably ambitious, but breadth is hard to achieve without some sacrifice of depth and so this is far from the definitive verdict on the Leaving Cert that you’d think from reports such as in the Irish Independent, where Katherine Donnelly writes “the findings present a case for radical change in the assessment process”.
Worryingly, the study seems to see knowledge not just as “lower order” but as actually inimical to the development of higher order skills. Here again we see the see-saw analogy that characterised the Junior Cycle reform: the fashionable idea that the reason many of our students struggle to be creative or to construct cohesive intellectual arguments is that they know too much, and it’s the fault of the pesky teachers who are expecting them to learn too much information. Following this line of thinking, the first thing we have to do if we want to foster higher order thinking is to reduce the amount of content that we expect young people to learn. We must also avoid at all cost the dreaded “rote-learning” and presumably focus more on inquiry methods and osmosis. The phrase “rote-learning” appears in all the media reports I have seen on this story and is taken to refer not just to learning essays by heart but to all attempts to memorise factual knowledge. The report criticises Biology in particular
“The very heavy focus on “factual knowledge” (73%) in Biology would raise questions as to the appropriateness of the subject as a basis for pursuing third level programmes in life sciences which emphasise the use of the scientific method”.
Elsewhere there is a nod to postmodernism that sees young people’s intellectual development as inevitably including “an awareness of relativism….knowledge is not certain, absolute truth but is contextualised and uncertain”. This distrust of factual knowledge and the desire instead to focus on generic skills of analysis and evaluation is described by Daisy Christodoulou as “educational formalism” and she writes in “Seven Myths about Education”
“…many of the myths work on the assumption that form is more important than substance. If I had to come up with an intellectual trend that underpins them, then I would choose postmodernism…Postmodernism is sceptical about the value of truth and knowledge, and many of these myths have their heart a deep scepticism about the value of knowledge.”
This “scepticism about the value of knowledge” runs through the Burns et al study and through the media reports, showing the growing prevalence of the myth that is one of Christodoulou’s top seven: facts prevent understanding. This myth could not be further from the truth. I think most teachers – even if unfamiliar with the ED Hirsch or Daniel Willingham – know intuitively that the opposite is true: far from being lower-order, the ability to recall, organise and utilise factual information is crucial to meaningful intellectual endeavour. Paul Kirschner and John Sweller, in their well-known paper on the inadequacies of discovery learning are unequivocal that the consensus among cognitive scientists is that “long-term memory is now viewed as the central dominant structure of human cognition. Everything we see, hear and think about is critically dependent on our long-term memory”. As Peps McCrea writes in this recent paper for the UK-based Institute of Teaching, “The more we know, the better we can think, and the better we think, the more we can know”.
***
The second section of the Burns et al study involves qualitative analysis of interviews with recent candidates. Two months after the exam the candidates were presented again with the paper and asked to talk through what they could remember of their thought process in the actual exam. This is an interesting and potentially fruitful way of carrying out the research. However, the evidence from these interviews is being wildly inflated and portrayed as definitive proof of teaching methods at senior cycle, and weaponised in an attack on the current examination system.
The sample size of the interviewees was 30 and they were drawn from a total of 10 schools. The students were a “convenience sample drawn through research contacts within the 10 schools.” Were the students randomly selected from the school rolls, or hand-picked by teachers known to the researchers? The researchers themselves freely admit
“with 19 of the 30 students from urban middle-class schools, there is a bias in the sample, which raises the question of how representative they were of the whole group of students who had just completed the leaving Certificate.” [italics mine]
They don’t give a date for when these interviews took place, but I am guessing it was after 2010 which would mean that the papers discussed in the interviews were not drawn from the same sample as the papers subjected to linguistic analysis.
The kind of analysis done in the study of the exam papers is a good start, but the papers themselves are only part of the process and the marking schemes are arguably more revealing, as are the Chief Examiners reports. Likewise, “stimulated recall” interviews could also be carried out with examiners and advising examiners, to get an idea of their thoughts when applying the schemes. While there is nothing new in the revelation that some candidates learn off essays in the hope of shoe-horning them in on the day, the study does not provide evidence that this is the most common method of preparation nor that it is endorsed and encouraged by teachers. And it offers no evidence that this method of revising leads to high grades: yes it is anecdotally associated with grind-schools, whose pupils often earn high grades, but there are many other variables around these candidates and it is possible that their exam success is in spite of rather than due to these methods. What about the vast majority of candidates – those who attend “normal” schools – and their preparation techniques and the relation of these techniques to the grades they achieve? We are not told how the candidate who learned off the 30 essays fared in the English exam, and this is a question I would need to see answered before I would join in any calls for the Leaving Cert to be abolished.
Candidates interpreted the “evaluate” questions as less rigorous, which is strange if they’re supposed to be higher-order. The implication was that you couldn’t be wrong if the majority of the marks were given for how well you made your case, and anecdotally I have often found that Leaving Cert English students often labour under the illusion that there are “no wrong answers” in English, even though there are such as the example of saying Boxer represents the bourgeoisie.
One area where there might appear to be no wrong answers is English Paper I. This exam paper comes in for considerable praise from the writers of the report. It’s creative. Candidates can demonstrate their higher order thinking skills on topics of their choosing, such as their own life. They can write short stories, somehow using no knowledge of literature whatsoever. It’s enjoyable and candidates are free of the anxiety of recalling quotations or trying to remember if Gertrude is married to Claudius or Polonius.
In the Irish Times article on the report Dr Burns explains that “They [the candidates] enjoyed when they could be creative for example in English Paper 1 when they are given instruction to write an essay. They expressed anxiety when they had to learn off and retain a huge body of text, which is a requirement for biology or geography.” This juxtaposition betrays two false assumptions about exams and assessment in general. The first is that experiential enjoyment is a desirable factor in assessment, and the second is that the anxiety felt by candidates who fear they might not have learned enough is a negative feature. I would say enjoyment is a neutral factor: exams should not be designed for the candidates’ pleasure, but neither should they be unnecessarily and deliberately discomforting. But the issue of anxiety is essential to exams: if you know for certain that you will succeed regardless of your mastery of the material then that’s not an exam, that’s a worksheet.
These are reported comments interviewees made in relation to Leaving Cert English.
“I really enjoyed writing that story with a ghostly presence.”
“I really enjoyed that…writing about moments of uncertainty in my life”
“I really liked writing my own thoughts in that essay. That’s why I like English so much more than other subjects”.
The authors note that these comments are “positive and confident” but these exam questions are not the best examples of the kind of higher order thinking of which that we wish our school-leavers to be capable. Take the candidate who wrote about “moments of uncertainty in his/her life”. Where is the required evidence that this candidate has learned anything useful in his/her time at school? These questions are seen as nice, in contrast to nasty exams that leave candidates “very concerned with being “wrong”” in a way that reminded me of a recent, excellent blog post that compares schooling to dentistry. We might praise a dentist because she does not inflict pain, but this does not necessarily make her an excellent dentist, and we might have been better off undergoing an unpleasant procedure elsewhere.
“It is worth noting that the exercise of a higher order skills (“evaluate”) is expressed as a positive experience by students.” An exam question involving “evaluation” is seen as being less likely to expose gaps in knowledge, misunderstandings and misinterpretations. They are, quite frankly, “bullshitable”. And bullshitting meets many of the characteristics of a higher-order skill, but again, is this the primary purpose of our education system?
A candidate who has rote-learned 30 essays is well-prepared only when the questions are vague, when these essays can be replicated no adjustments (or only minor ones) and when they are then are rewarded with higher marks than essays written using the candidate’s knowledge of the subject domain. The Leaving Cert Paper II Single Text questions have been headed in this direction for some time, although the Studied Poetry section continues to be quite specific. Then there is the Comparative Study section which is the epitome of Christodoulou’s point about the valuing of form over substance. Candidates have to compare three texts and marks are awarded for their skills in creating an answer that manages to weave the three into a coherent essay on the basis of a necessarily generic question. While this is no mean feat and few do it well, there is no room in the marking scheme to reward an answer that shows good understanding of Jane Austen’s “Persuasion” over one that a similar level of understanding of Stephen Chobsky’s “The Perks of Being a Wallflower”. Both of these are “texts” and as such merely vehicles for the fetished higher-order skills of evaluation and comparison.
***
The main problem I have is not with the study itself, which looks at important aspects of senior cycle and freely admits its own short-comings. I disagree with its conclusions but the authors themselves assert that their results are “rooted in interpretation” and therefore open to alternative readings. I agree with the authors that t over-reliance on pre-prepared answers is undesirable and that exams need to extend somewhat beyond purely Q&A responses (which most of the exams already do), but the remedy I see for this is not less memorisation but more of it, specifically a lot more of it at Junior Cycle and possibly also in primary.
What I see as the bigger issue here is how yet another limited study has been reported in the Irish media as conclusive proof that the Leaving Cert is broken. At least this one only half-relies on self-report, unlike last week’s offering (again from DCU). This narrative of a cruel, ineffective and out-dated senior cycle is already being repeated on a loop with the natural result that when the DES makes their plans for Senior Cycle reform public they will be welcomed as at the very least an improvement on what went before. We are being softened up and all the main media outlets are colluding in this process rather than interrogating the official line. Only when the last person who knows that you need factual knowledge to understand and interpret the world around you has retired, will we realise that you can’t critically think about nothing.
Kate Barry 