Education: the shift from qualitative to quantitative
Just make sure you tick those boxes!
Three years ago, a very bright child of my acquaintance managed to flunk an English exam at school. Among other things, the child did not follow the rubric for “how to write an argumentative essay”, which was extremely prescriptive. I was astonished by this, as the child can write very well—as the teacher apparently acknowledged—but the child fulfilled none of the prescriptive and rigid criteria for the assessment.
It is fair to say that hearing of this incident was a lightbulb moment for me. I had noticed for some years that students occasionally asked questions like, “Is there a particular sentence we should start our essay with?” I was bamboozled at first.
Now, I understood exactly why: students had been taught to write in this way at school, and their anxiety was understandable: the consequences for failing to tick the boxes in school had been drastic. At university, however, the last thing I want is a colour by numbers approach.
I began to wonder about how such a prescriptive approach had arisen in schools. It caused me to think back to how I had come up with the marking rubrics I now use regularly for exam papers. Over fifteen years ago, a student came to my office, determined to challenge their grade and receive a higher one. I spent an hour going through the paper with the student, with the top paper in the year level beside it for comparison.
After that, I developed a rubric, where I would tick off whether certain issues were considered or not. I don’t allocate marks to each point, and my assessment of exam papers is still ultimately qualitative. One can discuss every issue in a pedestrian manner, or miss an issue, but write an exceptional and perceptive paper in all other regards.
I wonder if the more quantitative approach to assessment has been adopted in part because of students or parents mounting challenges like the one I relate above. It creates a natural move to a more quantitative mode of assessment, because it’s harder to argue against when there has been little exercise of discretion, and the rules are very rigid.
Rubrics also have the advantage of producing certain criteria, assuring a level of consistency. During my law degree, it was not always clear why I got the grade I did; if the criteria are clear, students can learn how to improve. Rubrics also ensure all markers are marking on the same basis.
There is a place for rubrics and criteria, but we have to be careful to balance the need for clear expectations with an assessment of quality and an allowance for creative solutions for a problem which may not fulfil all the criteria, but which are nonetheless quality answers.
There is a danger, however, that a very rigid quantitative approach will stifle creativity, and encourage an algorithmic approach to academic work: if you tick these boxes, that is enough.
When I was studying GCSE Maths, many moons ago, we were given an assignment where we had to construct a box for ping-pong balls. Upon adult reflection, the point of the assignment was to teach us about cube nets.
No cube nets for me! I came up with an idea involving folding origami boxes (complete with mathematical measurements of the squares and triangles involved). I wonder if, in the current day and age, I would have flunked the assignment, like the child at the start of this post. The assessment was clearly designed to make us write about cube nets. My solution did not involve cube nets at all, although I did explain the maths of it. Luckily for me, I was marked on a qualitative basis, and I was not punished for the lack of discussion of cube nets, but rewarded for “thinking outside the box.” (Sorry-not-sorry.)
It strikes me that this difficulty is something we face in law as well. We have to make laws certain enough that people know what rules they have to follow, but not so rigid that the law becomes unfair. There are several facets to this. First legal scholars distinguish between rules and standards,1 although in practice the distinction is more a spectrum than a simple binary.2 Rules tend to be determinate and give the decision maker less discretion. By contrast, standards tend to give decision makers more discretion and allow decision makers to take account of the background justifications for a law, in the light of the specific circumstances.3
A rule has two parts. First, the rule outlines the factual conditions which trigger its application. Secondly the consequence prescribes what happens if the conditions are triggered.4
The example often used of a legal rule is a traffic speed rule. For example, there may be a rule that drivers of motor vehicles must not drive faster than 60 kilometres per hour in certain zones, and if they do, they are liable for a fine of $185. Driving above 60 kilometres an hour in those zones triggers the rule, and in this example, the consequence is also set out in the rule itself (the payment of a fine). A rule is knowable in advance: it states what conduct is allowed prior to that conduct taking place.5
A legal standard governing driving safety might stipulate that drivers must not use motor vehicles in a way which is ‘unreasonable and unsafe’. It would be left to the decision maker to decide whether the driver’s conduct had been unreasonable and unsafe in the specific circumstances.
There is also a distinction between generality and particularity, regardless of whether a law is expressed as a rule or a standard. A general rule might appear to be more certain, and minimises the opportunities for the decision maker to make their own rules. However, the more general a law is, the more it is likely to be both under- and over-inclusive, because it will catch some people it should not, and let go others it should not.
Frederick Schauer has argued that general rules are useful but could cause problems when ‘recalcitrant experiences’ undermine the generalisation. He outlines three ways in which recalcitrant experiences may arise:
A probabilistic generalisation is incorrect in the case before the decision-maker;
A supposedly universal generalisation turns out not to be universal; and
A fact suppressed by the generalisation may become relevant, but we are unable to access it because of the entrenchment of the rule does not allow us to refer to it.6
I promise you, there’s method to my madness, applying legal scholarship to the problem of student marking.
Returning to the example of the failed English essay, it seemed to me that the general rules for marking the exam were too rigid, and too particular. So the generalisation made by the marker might be “A good academic essay follows the TEEEL structure.” What is the TEEEL structure, you might ask? I had managed to make it all the way to Professor without being aware of this. TEEEL stands for:
Topic sentence: “This post will discuss the problems of overly rigid rubrics in student evaluation.”
Explanation: “There is a risk that an overly rigid use of rubrics will stifle creativity.”
Evidence and Examples: “Legal scholarship shows that sometimes, the generalisations behind the rule which comprises a rubric can give rise to ‘recalcitrant experiences’ (see Schauer, 1991). For example, it may be that an essay written in a very formulaic way is not easy to read, and does not really test the skills we are trying to teach and encourage.”
Link: “Therefore, overly rigid rubrics should be avoided, because of their propensity to make academic writing very boring, and to stifle creativity.”
In the case of the student who failed the exam, the probabilistic generalisation was wrong. A failure to follow the TEEEL structure did not mean that the student was not skilled in writing or using English language. Moreover, the supposedly universal generalisation: “Good essays rigidly follow a TEEEL structure” is not universal. Many good essays do not rigidly follow the TEEEL structure. In fact, a TEEEL structure makes the essay rather pedestrian and boring: it takes all the colour out of the writing. Finally, an insistence on a particular structure as an indicator of quality means that the teacher cannot look behind the criteria to such matters as, has the student understood the material he or she was supposed to read and comprehend? Has the student considered the material in a nuanced way and written about it in an insightful and grammatical way? The very things we should be looking for must be ignored.
Consequently, I think we must be very wary of extremely rigid and particular rules such as the one I’ve described above. It’s a good rule of thumb for a student who is struggling to know how to put together an argument, but it is totally unnecessary for a good argumentative essay to follow this structure slavishly.
We have to ask ourselves: what are the purposes of these rules? Are they simply there to assign grades in a certain and formalistic manner, or are they designed to help teach kids to read and write properly? Frankly, as a university lecturer, I’d prefer the latter. Sure, it’s necessary to have some guidance around how to write an essay, but there has to be some discretion, in the nature of a standard, to allow for the assessment of quality, not just the ability to tick a box. Otherwise, we may as well let Chat GPT take over.
D Kennedy, ‘Form and Substance in Private Law Adjudication’ (1976) 89 Harvard LR 1685; P Schlag, ‘Rules and Standards’ (1985) 33 UCLA LR 379; KM Sullivan, ‘Foreword: The Justices of Rules and Standards’ (1992) 106 Harvard LR 22; L Kaplow, ‘Rules Versus Standards: An Economic Analysis’ (1992) 42 Duke LJ 557; L Alexander and E Sherwin, ‘The Deceptive Nature of Rules’ (1994) 142 University of Pennsylvania LR 1191; C Sunstein, ‘Problems with Rules’ (1995) 83 California LR 953; R Korobkin, ‘Behavioural Analysis and Legal Form: Rules vs. Standards Revisited’ (2000) 79 Oregon LR 23; F Schauer, ‘The Convergence of Rules and Standards’ [2003] NZLR 303.
Sunstein, ibid, 961; Korobkin, ibid, 26
Sullivan (n 1) 58–59; Schauer (n 1) 309–11.
F Schauer, Playing by the Rules – A Philosophical Examination of Rule-Based Decision-Making in Law and in Life (1991) 23.
Kaplow (n 1) 559; Sunstein (n 1) 961–962.
Schauer (n 4) 45–47.
Another post I really enjoyed, Katy! And that you cited an article by Duncan Kenned- bonus for me. His work on "Training for hierarchy" has been my favourite piece of academic writing since uni days.