III. Ordinary Things Essay 18
The Exam Paper
One page, one number, and everything in between
A student hands in a paper. It is one page, or six. It has a number on the front, and the number is going to be the only thing anybody in a large building knows about this piece of work for the next two years. Somewhere behind the number is a piece of writing. It took four hours. It has a thing in the third paragraph that the writer did not plan, which is the most interesting thing on the page, and the number is going to be identical for a page with that thing in it and a page without it.
This is not a complaint about exams. It is a description of what an exam is, and the description is a bit strange: a compression of a piece of learning into a single number, made by somebody who was not there, on a day that is not like the day the learning happened.
A mark is an abstraction, and the whole argument about teaching to the test follows from that. A mark is a lossy compression of a piece of learning, and every lossy compression has a shape, and when the compression becomes the target, the shape starts to be what is produced. The students do not learn the subject. They learn the shape of what the mark can see.
Take a piece of school history teaching, about a period nobody finds interesting, for a class of students who are not going to do anything with this afterwards. The teacher has a choice. She can teach the period properly, which takes two terms, and which the students will not engage with, and the results will be a class full of students who can talk about the period and who will forget it. Or she can teach the twelve facts that appear every year on the paper, and the students will learn them, and the students will score well.
Now the interesting part. The twelve facts are not a summary of the period. They are a compression of it, made by whoever wrote the paper, for the purpose of producing a number that can be compared across a cohort. That purpose is not the same as learning history. The compression dropped everything that cannot be examined in forty minutes on paper, and what it dropped is most of what history actually is: the arguments, the disagreements between historians, the reasons a thing happened rather than the sequence in which it did.
Now change the condition. Suppose the teacher teaches the period properly anyway, in two terms, and the students engage because the teacher is good, and the students develop arguments and disagreements and reasons. And then they sit the exam, and the exam asks for the twelve facts, and some of them remember the facts and some do not, and the number does not reflect the argument they could have made. Which teacher was right? The one who taught the facts, if the number is the only thing anybody counts. The one who taught the arguments, if learning is what is being counted. The number cannot tell you, because the number does not contain the arguments.
What a mark is for
A mark is an abstraction with a job, and the job is comparison. A teacher has thirty papers and a limited amount of time. A department has a cohort and a number of places. A parent has one child and wants to know how the child is doing. In every case the number answers a question about standing, and standing is a comparison, and comparison needs a scale.
And the mark is a good abstraction for that job, within its scope. It is fast, it is cheap, it can be added up, it can be ranked, it can be passed around, and it can be compared between schools and between years. That is a genuinely valuable piece of machinery, and it is worth being clear that the objection to exams is not an objection to numbers. The objection is to the leap from a number being a useful summary to a number being the thing.
What a mark is for
The scope of a mark is the scope of the thing that was compressed. A mark out of ten on a history paper measures a piece of history writing. It does not measure curiosity, and it does not measure whether the student will keep reading after the course ends, and it very obviously does not measure how much the teacher taught. A mark that is used as evidence for any of those is being used outside its scope, and using a measurement outside its scope is the single most common error in the whole enterprise of measurement.
Teaching to the test
Teaching to the test is normally used as an accusation, and it deserves a more careful treatment, because there are two different things people mean by it and only one of them is a real problem.
The first is aligning teaching with what the test measures. If a test asks about causes and the teaching only covered a chronology, then the teaching has a gap and the test is right. Aligning is a good idea and it is what a sensible exam does by design.
The second is teaching the shape of the mark. This is the one that is a problem, and it happens automatically without anybody deciding to do it, and the mechanism is worth spelling out.
Here is the loop. The test measures some things. The teaching, over several terms, responds to what the test measures, because the teaching is done by people who know what the test measures. The practice does the same. Then the test comes round again, and now the things that are easy to mark are better taught, and the things that are hard to mark are less taught, and the next version of the test, written by people looking at the same evidence, drifts further towards the easy-to-mark things, because those are the ones the cohort did well on.
Nobody in this loop is doing anything wrong. Every step is a rational response to the step before. And the loop is reinforcing, so the drift compounds. A department that taught the twelve facts in 1990 and the twelve facts in 2020 is probably teaching slightly different twelve facts, and the difference between the two sets tells you almost nothing about history and everything about the marking.
What if the mark went away and every piece of work got a comment instead? The teacher's time does not change, because the teacher was reading the work anyway. What changes is the shape of the teacher's attention. A comment has to be about something specific in this piece of work, and a specific comment is the teacher telling this student what they did and what they might do. A number can be produced without the teacher having formed any view at all, and in a busy week, a number is what gets produced.
Now change the condition. What if the school still needed the number, for the cohort comparison and the league table, and simply added the comments on top? The number would still compress. The teacher would still be producing it. And the comments would still be there, on this paper, for this student, in a form that the number could not have carried. The expensive thing about marks is not the marking. It is the attention that goes nowhere once the mark has been made.
"Marks are objective." They are not. A mark is a judgement made by one person about a piece of work, using a scale, with a rubric that was written by somebody else, and with a different person marking the same paper would produce a different number. The number looks precise because it is expressed in a scale, and the precision is in the scale rather than in the judgement. That is fine. Plenty of useful things are imprecise measurements of something real. What is not fine is treating the number as if the judgement had not happened.
The other misconception: "exams measure what matters, so there is no issue." The test is a measurement with a scope, and the scope is narrow by necessity, because a test has to be short, written down, and marked by a stranger. The narrowness is a feature of the instrument. The mistake is assuming the instrument measures the whole of what it is aimed at, and then acting as though it does.
Fairness
There is a separate and sharper question about the mark, which is whether the process is fair, and it is worth keeping separate from the question of what a mark measures, because the two get confused constantly.
Two things make an assessment unfair, and they are quite different. The first is that the measurement does not capture what it is meant to capture. A student who understands the argument but cannot fit it on the page in the time allowed is poorly measured, and no amount of fairness procedure fixes that.
The second is that the situation around the test is not the same for everybody. Some students have a quiet room, some do not. Some have done the reading, some have not, and the gap between them is not about ability. Some have a teacher who has taught the twelve facts, and some have a teacher who has taught the period. A test run in a school where every teacher has aligned to the same compression will produce a fairer distribution of numbers than a test run in a school where each teacher has taught what they thought mattered, and the fairness has been bought by a narrowing that nobody examined.
That is the uncomfortable finding. Improving the fairness of a measurement can be done by improving the measurement or by making everybody's teaching more alike, and the second is much easier and much more damaging.
Limits
Nothing in this essay is an argument against testing. Testing is how anybody finds out whether a student has learned a thing they said they had learned, and the alternative, which is not testing, is taking everybody's word for it, which is how entire cohorts of people arrive at a workplace unable to do the job they were hired for.
Second limit: the argument is about marks as summaries of learning, and there is a much older and better tradition of assessment that this site has barely touched. A portfolio, a piece of work made over a term, a conversation, a demonstration, a defence of your own position against somebody who disagrees. These compress less and they cost more, and the reason they cost more is the very thing that makes them better, which is that somebody has to know what the student was trying to do.
Third limit: a mark is not always a compression of learning. Sometimes it is a compression of a skill, and skills compress well. Typing speed, a times table, a knot, a swimming stroke. These are exactly the things where a short test is a good instrument, because the thing being measured is bounded, describable, and reliably demonstrated. The problem with marks is not the number. It is the number being asked to carry something unbounded.
Stuck
You are stuck at the abstract level with an exam when you are looking at the numbers. The distribution is interesting. The average is moving. The gap between the two schools is narrowing. And you have no idea what any of it means, because you have been looking at the compression and not at the thing that was compressed.
The way out is to go and read three scripts. Not to find the errors. To read them, and to see what the students were trying to do, and to notice that one of the three contains an argument you could not have predicted from the mark it got. Then write one sentence about the exam that the mark cannot say. If you can write that sentence, the test has a purpose that the number does not carry, and that purpose is the thing worth defending, and it is also the thing that the loop will erode if nobody holds it.
Sources
- Goodhart's law, named after Charles Goodhart, is the usual reference point for a measure becoming a target. The wording used here is a paraphrase, and the point is discussed at more length in an earlier essay on this site.
- No specific exam board, qualification, marking scheme, or study is cited in this essay, and the school history example is hypothetical.
- The argument that narrowing teaching can improve measured fairness is offered as an observation about how the system works, not as a finding from any particular research.