Another great post, Katy. Teaching contract law has made me realise how few textbooks are truly written with student learning in mind. I still rely heavily on McKendrick’s Contract Law, but recently noticed he’s now Emeritus-which made me wonder: who will take up the mantle next? We desperately need authors who can write with clarity and accessibility, not just for students but also for early-career academics. The incentives, of course, are skewed as in many professions, the more socially useful the work, the less it's rewarded. Maybe writing a good textbook is a bit like being a schoolteacher: vital to the system, but undervalued.
“Academics must also recall that we are custodians of knowledge. “
Hmmm. That might have been true in the past, but is it true today? If I want to understand the preconditions for contract formation, Google is my friend. If I want to evaluate proposed contractual departures by a tenderer, AI will give me a reasoned critique of the specific terms and the risks implied in minutes.
So what knowledge do academics curate that I need, but can’t access without them? Obviously, if I want to be a lawyer I have to study, but given the unbelievable bias among University law lecturers highlighted by Janet Albrechtsen among others, why would anyone expose themselves to that?
My experience is that kids are disinterested in information in advance of the need for its application. “Why do I need to know this?” The appeal of information on demand when it is needed is irresistible. AI may have bias, but is the bias less than that of activist zealots at Universities who use their professional role as a platform for their personal beliefs?
So the thing that upsets me about those newspaper reports is that people then think we’re all that way, and we’re not. I hope you can tell from my writing that I value knowledge and I do not try to push my views on students. In fact, there’s nothing I would like less, because it would be boring to just get my views repeated back at me. Many of my colleagues are like me, just less forthright about it.
That being said, there are problems with the academy, and I have written about them extensively. As I have said repeatedly, I think activism is incompatible with proper scholarship. And it has no place in the classroom, in my view.
Re “Why do I need to know this?” - that is where that teaching course came in really useful. It taught me of the need to make expectations clear at the outset and tell students exactly why they had to know things. I am very proud to say that sometimes I get students contacting me years later to say, “You know, that was useful…” My job is done.
There is a scaling issue at play here. When research universities were first created, they were relatively elite organizations with two goals: research and education, at a time when the natural sciences were relatively unexplored, meaning that there was plenty of scientific research to be done. It was reasonable under those circumstances that a measure of success for researchers be the papers they were able to publish. The universities added liberal arts to their curriculum because their students needed them (or at least the ability to write reasonably well), and applied the same criteria, even though it surely made less sense in those areas.
But as the easiest areas in science were plumbed, it became harder and harder to produce innovative research - and when we decided to send massive numbers of students to the universities, that required even more professors trying to write such papers.
Meanwhile, there were few if any measures for teaching competency. I was very fortunate to have two brilliant teachers in college in difficult subjects (organic chemistry and quantum mechanics) but my stat mech prof was an almost totally useless teacher. If you hire people for their competence in research, you have no reason to expect them to be good teachers as well. When you don't need many of them, you can probably focus on the best ones, but as the number of professors needed grows, that's harder.
Yes, this is all very true. So many pressures forcing the universities to be like this. It’s very difficult to measure teaching competence. Student surveys more often measure popularity, unless carefully designed. Students’ results depend in part on their own effort. The way we’ve started to measure teaching competence is to do peer review - get an experienced teacher to sit in and give detailed feedback. I am that rare beast, someone who teaches what I research and genuinely enjoys teaching and finds that the teaching feeds back into her research. But I think I’m a vanishing minority.
AIs may herald the return of a more "scholarly" model of education as they take over entirely the basic transmission of core knowledge and skills, and the human teachers are reserved for guiding essentially research-oriented students through their own paths.
It will be interesting to see what happens with AI. It’s a huge problem: students getting AI to write essays for them. The thing is, that doesn’t teach you anything. The reason I write this blog is to work out what I think. It’s through writing that I figure things out and learn. Prediction: much more of a focus on handwritten closed-book exams and viva voce examinations.
Indeed! Which also suggests a 'return to the past' with higher education as we now conceive it returning to a rarefied thing reserved to a small (hopefully cognitive not cultural) elite, and most people going to (more bureaucratic!) expanded professional or trade schools, which is what higher education as we currently practice it is well down the road to being. In my mind, this split is the only way we have to resolve the current tension between higher education as we conceive of it and as we practice it, and it is long overdue!
In law, I'd really dispute that that transmission of core knowledge and skills is 'basic' or just a task for AI. You might use AI to reinforce learning of principles and what is in a case, but that's just a small piece of the puzzle.
I think we need to take seriously how hard it is to clearly define what a lawyer does, what skills they use, what it is that makes a lawyer a good lawyer. The process of becoming a good lawyer is so layered, iterative, experiential, based on relationships with mentors, it's something very organic and hard to describe. That iterative process starts with your relationship with your law lecturers and the conversations you have with your fellow students. Imparting the information is not enough - it's the relationships in which it's embedded that really matter.
I agree with you about most of that! Indeed, after a few years of practicing law I wondered if law was not better taught more like a trade apprenticeship. I still think that.
I don't agree that we need to clearly define it to teach it, try asking a plumber for an academically satisfying definition of what s/he does for a job.
I do think that LLMs can easily teach at least 2/3rds of the content of a law degree, and I strongly suspect more. But please don't underestimate how little a surprisingly high number of students actually learn!
And I'm not suggesting that students don't be in cohorts and encouraged to talk (and work!) together, I'd definitely want them doing that, and ideally I would have law faculty as mentors meeting with small groups (say max 20) 2 or 3 times a week, but I'd probably make it just once a week in the first year, and maybe the second too.
I agree with you on Goodhart's law and there is no shortage of total trash in academia as a result of arbitrary KPIs. However KPIs do mitigate the fact that in their absence, many hiring or grant decisions may be left to subjective judgment, and that is often a proxy for whoever has the strongest relationship with the person doing the hiring or making the grant. It also leaves more room for outright bias of those in power propelling those who agree with them or whose ideas are compatible with theirs to the top. Now, these biases will exist no matter what but KPIs dilute them.
If there could be a system where the party in charge of hiring / granting could be simultaneously disinterested in the decision and suitably qualified to make the decision, then I would call that progress of a kind. The only way I see that happening though is through stringent conflict checks and reciprocal arrangements where hiring/grant decisions about one particular academic / position / grant must be made by a person that is unaffiliated with the institution and has cleared a stringent conflicts check, though I fear this may cause more problems than it solves.
This is also not to say that number of publications / citations etc is not a useful metric. It should be part of the equation, but I agree it should not be the only metric or determinative.
Noam, thanks so much, this is a great defence of the KPIs. I can see I am going to have to write a follow up post to show why they actually don’t work and have perverse incentives, particularly if you write on a pretty narrow area. Moreover the subjective judgment and the old school tie considerations are still VERY VERY operative with who gets cited and who gets approved. It actually promotes group think and an unwillingness to question. Anyway, thank you for pushing back, because this was the best possible defence of the current system I can think of - makes me think more.
I look forward to reading it. I have no doubt they have perverse incentives - I guess my issue is whether some alternate hypothesis is an improvement on the situation. One way to explore it is also to work through whether KPIs are even used as intended. I know from my own work that they are not. Decision makers usually have already made the decision up in their minds and they use (or downplay) KPIs as needed to get to where they want to go - so arguably they are a redundant step.
Thanks for these thoughts, as interesting as ever!
I am not sure, however, that the phenomenon that you describe can be attributed solely, perhaps not even mainly, to post-WWII funding pressures.
It appears to me to be part of a much more secular trend that one can describe in many ways, but which amounts to the reification and formalisation of previously implicit norms. This is hugely beneficial for those who are not members of the social and cultural elite, it is self-evidently fairer. Further, in practice, the benefits to them have been sufficiently widely distributed that this has doubtlessly contributed to the vastly richer and more prosperous societies that we have today.
It is not, of course, a free lunch. We are doubtlessly also poorer than we might have been because of this, since there appears to be good reason to think that simply “funding geniuses” remains the best way to do a lot of science, and not just science (consider Wittgenstein, Lewis, and Tolkien: would these men have been able to contribute what they did to our worlds without the almost wholly sheltered from external pressures life of the “Oxbridge Don”?).
That is not to say that we are poorer than we would have been in the hypothetical case in which we retained the informal and highly culturally mediated norms of the past: it is entirely possible that this is a case of one or the other but not both.
Perhaps the ideal synthesis is a handful of unashamedly elite institutions in a sea of formalised relative mediocrity (which can still be a very high standard and can still deliver massive scientific and social value!). But for this the institutions in question have to believe in the value of their own exceptionalism, a belief which may require more analytical thinking ability (conscientiousness + disagreeability) than their management has selected for these past few decades.
Or perhaps I am just idealising a past that we can never return to and should not want to.
These are great thoughts, Delenda Est. Relatedly, I keep thinking that we have been moving from a qualitative to a quantitative analysis of quality over decades. Whether something is good or not depends upon how many boxes you ticked correctly. I think it’s all part of the same phenomenon. Perhaps it has to do with the ever encroaching reach of bureaucracies. Perhaps it also has to do with increasing technology. Computers are binary. It’s all 0s and 1s, and nothing in between. This means a lack of flexibility. If something is not on the drop down menu, then it can’t be accommodated.
I think there were certain benefits from the past way in which university operated, as well as detriments. As Thomas Sowell says, it’s trade offs, all the way down. What we have to work out is - have the incentives we’ve imposed actually encouraged some really bad practices in some instances? I’ve noticed the increasing distrust in academic research. The failure of various studies to replicate seems to back this up. Also some of the cultural studies stuff I see on social media seems like a waste of time to this hard-headed lawyer. So, whatever happens, I think we have to change, or lose our social licence.
Amen to Thomas Sowell's comment about tradeoffs, to which one could add incentives!!
Without much evidence, my suspicion is that the growth of bureaucracy is a symptom of the same trend, not the virus itself. But I think it did have the way for the DEI mutation of the virus.
Digitalisation is trickier, and I would be quite surprised if it did not at least act as an accelerant. On that front, there may be light ahead: AIs can make much more subtle and multifactorial judgements, so, assuming that they aren't just doing the whole job and assuming that there is still any point educating humans, AI administrators may be a significant improvement.
Also, AIs may herald a return to a more "scholarly" model of learning as
Sometimes I wonder whether a relentless pursuit of something "new" just results in obfuscation and complexity whereas genuinely new revelations can often be obtained by attempting to reach a position of clarity on something that is widely believed to be well-understood but, on closer inspection, isn't. Unfortunately, we rarely seem to appreciate the value in something that is clear and simple, even though it is usually far harder to produce something clear and simple than it is to produce something "new" and complex.
I also wonder whether, in an AI future and the access it appears to provide to collective group-think on a massive scale, we will value textbooks more for pointing out the errors in the conclusions that AI might be trawling from the decisions or whether we will come to accept that the law is whatever AI says that it is.
I do love reading your writing about perverse incentives in academia, Katy. The law too is a vast collection of incentives, many of which have a perverse operation!
I was cheeky, and sent it to the production editor of the ALJ. It could do with a wide audience.
Teehee. They know me well.
Another great post, Katy. Teaching contract law has made me realise how few textbooks are truly written with student learning in mind. I still rely heavily on McKendrick’s Contract Law, but recently noticed he’s now Emeritus-which made me wonder: who will take up the mantle next? We desperately need authors who can write with clarity and accessibility, not just for students but also for early-career academics. The incentives, of course, are skewed as in many professions, the more socially useful the work, the less it's rewarded. Maybe writing a good textbook is a bit like being a schoolteacher: vital to the system, but undervalued.
Pretty much. McKendrick is great, isn’t he?
“Academics must also recall that we are custodians of knowledge. “
Hmmm. That might have been true in the past, but is it true today? If I want to understand the preconditions for contract formation, Google is my friend. If I want to evaluate proposed contractual departures by a tenderer, AI will give me a reasoned critique of the specific terms and the risks implied in minutes.
So what knowledge do academics curate that I need, but can’t access without them? Obviously, if I want to be a lawyer I have to study, but given the unbelievable bias among University law lecturers highlighted by Janet Albrechtsen among others, why would anyone expose themselves to that?
My experience is that kids are disinterested in information in advance of the need for its application. “Why do I need to know this?” The appeal of information on demand when it is needed is irresistible. AI may have bias, but is the bias less than that of activist zealots at Universities who use their professional role as a platform for their personal beliefs?
So the thing that upsets me about those newspaper reports is that people then think we’re all that way, and we’re not. I hope you can tell from my writing that I value knowledge and I do not try to push my views on students. In fact, there’s nothing I would like less, because it would be boring to just get my views repeated back at me. Many of my colleagues are like me, just less forthright about it.
That being said, there are problems with the academy, and I have written about them extensively. As I have said repeatedly, I think activism is incompatible with proper scholarship. And it has no place in the classroom, in my view.
Re “Why do I need to know this?” - that is where that teaching course came in really useful. It taught me of the need to make expectations clear at the outset and tell students exactly why they had to know things. I am very proud to say that sometimes I get students contacting me years later to say, “You know, that was useful…” My job is done.
There is a scaling issue at play here. When research universities were first created, they were relatively elite organizations with two goals: research and education, at a time when the natural sciences were relatively unexplored, meaning that there was plenty of scientific research to be done. It was reasonable under those circumstances that a measure of success for researchers be the papers they were able to publish. The universities added liberal arts to their curriculum because their students needed them (or at least the ability to write reasonably well), and applied the same criteria, even though it surely made less sense in those areas.
But as the easiest areas in science were plumbed, it became harder and harder to produce innovative research - and when we decided to send massive numbers of students to the universities, that required even more professors trying to write such papers.
Meanwhile, there were few if any measures for teaching competency. I was very fortunate to have two brilliant teachers in college in difficult subjects (organic chemistry and quantum mechanics) but my stat mech prof was an almost totally useless teacher. If you hire people for their competence in research, you have no reason to expect them to be good teachers as well. When you don't need many of them, you can probably focus on the best ones, but as the number of professors needed grows, that's harder.
Yes, this is all very true. So many pressures forcing the universities to be like this. It’s very difficult to measure teaching competence. Student surveys more often measure popularity, unless carefully designed. Students’ results depend in part on their own effort. The way we’ve started to measure teaching competence is to do peer review - get an experienced teacher to sit in and give detailed feedback. I am that rare beast, someone who teaches what I research and genuinely enjoys teaching and finds that the teaching feeds back into her research. But I think I’m a vanishing minority.
Oups premature submission.
AIs may herald the return of a more "scholarly" model of education as they take over entirely the basic transmission of core knowledge and skills, and the human teachers are reserved for guiding essentially research-oriented students through their own paths.
It will be interesting to see what happens with AI. It’s a huge problem: students getting AI to write essays for them. The thing is, that doesn’t teach you anything. The reason I write this blog is to work out what I think. It’s through writing that I figure things out and learn. Prediction: much more of a focus on handwritten closed-book exams and viva voce examinations.
Indeed! Which also suggests a 'return to the past' with higher education as we now conceive it returning to a rarefied thing reserved to a small (hopefully cognitive not cultural) elite, and most people going to (more bureaucratic!) expanded professional or trade schools, which is what higher education as we currently practice it is well down the road to being. In my mind, this split is the only way we have to resolve the current tension between higher education as we conceive of it and as we practice it, and it is long overdue!
In law, I'd really dispute that that transmission of core knowledge and skills is 'basic' or just a task for AI. You might use AI to reinforce learning of principles and what is in a case, but that's just a small piece of the puzzle.
I think we need to take seriously how hard it is to clearly define what a lawyer does, what skills they use, what it is that makes a lawyer a good lawyer. The process of becoming a good lawyer is so layered, iterative, experiential, based on relationships with mentors, it's something very organic and hard to describe. That iterative process starts with your relationship with your law lecturers and the conversations you have with your fellow students. Imparting the information is not enough - it's the relationships in which it's embedded that really matter.
I agree with you about most of that! Indeed, after a few years of practicing law I wondered if law was not better taught more like a trade apprenticeship. I still think that.
I don't agree that we need to clearly define it to teach it, try asking a plumber for an academically satisfying definition of what s/he does for a job.
I do think that LLMs can easily teach at least 2/3rds of the content of a law degree, and I strongly suspect more. But please don't underestimate how little a surprisingly high number of students actually learn!
And I'm not suggesting that students don't be in cohorts and encouraged to talk (and work!) together, I'd definitely want them doing that, and ideally I would have law faculty as mentors meeting with small groups (say max 20) 2 or 3 times a week, but I'd probably make it just once a week in the first year, and maybe the second too.
I agree with you on Goodhart's law and there is no shortage of total trash in academia as a result of arbitrary KPIs. However KPIs do mitigate the fact that in their absence, many hiring or grant decisions may be left to subjective judgment, and that is often a proxy for whoever has the strongest relationship with the person doing the hiring or making the grant. It also leaves more room for outright bias of those in power propelling those who agree with them or whose ideas are compatible with theirs to the top. Now, these biases will exist no matter what but KPIs dilute them.
If there could be a system where the party in charge of hiring / granting could be simultaneously disinterested in the decision and suitably qualified to make the decision, then I would call that progress of a kind. The only way I see that happening though is through stringent conflict checks and reciprocal arrangements where hiring/grant decisions about one particular academic / position / grant must be made by a person that is unaffiliated with the institution and has cleared a stringent conflicts check, though I fear this may cause more problems than it solves.
This is also not to say that number of publications / citations etc is not a useful metric. It should be part of the equation, but I agree it should not be the only metric or determinative.
Noam, thanks so much, this is a great defence of the KPIs. I can see I am going to have to write a follow up post to show why they actually don’t work and have perverse incentives, particularly if you write on a pretty narrow area. Moreover the subjective judgment and the old school tie considerations are still VERY VERY operative with who gets cited and who gets approved. It actually promotes group think and an unwillingness to question. Anyway, thank you for pushing back, because this was the best possible defence of the current system I can think of - makes me think more.
I look forward to reading it. I have no doubt they have perverse incentives - I guess my issue is whether some alternate hypothesis is an improvement on the situation. One way to explore it is also to work through whether KPIs are even used as intended. I know from my own work that they are not. Decision makers usually have already made the decision up in their minds and they use (or downplay) KPIs as needed to get to where they want to go - so arguably they are a redundant step.
Thanks for these thoughts, as interesting as ever!
I am not sure, however, that the phenomenon that you describe can be attributed solely, perhaps not even mainly, to post-WWII funding pressures.
It appears to me to be part of a much more secular trend that one can describe in many ways, but which amounts to the reification and formalisation of previously implicit norms. This is hugely beneficial for those who are not members of the social and cultural elite, it is self-evidently fairer. Further, in practice, the benefits to them have been sufficiently widely distributed that this has doubtlessly contributed to the vastly richer and more prosperous societies that we have today.
It is not, of course, a free lunch. We are doubtlessly also poorer than we might have been because of this, since there appears to be good reason to think that simply “funding geniuses” remains the best way to do a lot of science, and not just science (consider Wittgenstein, Lewis, and Tolkien: would these men have been able to contribute what they did to our worlds without the almost wholly sheltered from external pressures life of the “Oxbridge Don”?).
That is not to say that we are poorer than we would have been in the hypothetical case in which we retained the informal and highly culturally mediated norms of the past: it is entirely possible that this is a case of one or the other but not both.
Perhaps the ideal synthesis is a handful of unashamedly elite institutions in a sea of formalised relative mediocrity (which can still be a very high standard and can still deliver massive scientific and social value!). But for this the institutions in question have to believe in the value of their own exceptionalism, a belief which may require more analytical thinking ability (conscientiousness + disagreeability) than their management has selected for these past few decades.
Or perhaps I am just idealising a past that we can never return to and should not want to.
Some related thoughts here: https://marginalrevolution.com/marginalrevolution/2022/07/science-as-a-source-of-social-alpha.html
https://marginalrevolution.com/marginalrevolution/2024/10/podcast-on-science-policy.html
https://johnhcochrane.blogspot.com/2021/06/fast-grants-and-economics-of.html (this is in some ways a more detailed version of my argument above, but with a different focus)
These are great thoughts, Delenda Est. Relatedly, I keep thinking that we have been moving from a qualitative to a quantitative analysis of quality over decades. Whether something is good or not depends upon how many boxes you ticked correctly. I think it’s all part of the same phenomenon. Perhaps it has to do with the ever encroaching reach of bureaucracies. Perhaps it also has to do with increasing technology. Computers are binary. It’s all 0s and 1s, and nothing in between. This means a lack of flexibility. If something is not on the drop down menu, then it can’t be accommodated.
I think there were certain benefits from the past way in which university operated, as well as detriments. As Thomas Sowell says, it’s trade offs, all the way down. What we have to work out is - have the incentives we’ve imposed actually encouraged some really bad practices in some instances? I’ve noticed the increasing distrust in academic research. The failure of various studies to replicate seems to back this up. Also some of the cultural studies stuff I see on social media seems like a waste of time to this hard-headed lawyer. So, whatever happens, I think we have to change, or lose our social licence.
Amen to Thomas Sowell's comment about tradeoffs, to which one could add incentives!!
Without much evidence, my suspicion is that the growth of bureaucracy is a symptom of the same trend, not the virus itself. But I think it did have the way for the DEI mutation of the virus.
Digitalisation is trickier, and I would be quite surprised if it did not at least act as an accelerant. On that front, there may be light ahead: AIs can make much more subtle and multifactorial judgements, so, assuming that they aren't just doing the whole job and assuming that there is still any point educating humans, AI administrators may be a significant improvement.
Also, AIs may herald a return to a more "scholarly" model of learning as
Thoughtful, and valuable insight, Katy.
Thank you.
Sometimes I wonder whether a relentless pursuit of something "new" just results in obfuscation and complexity whereas genuinely new revelations can often be obtained by attempting to reach a position of clarity on something that is widely believed to be well-understood but, on closer inspection, isn't. Unfortunately, we rarely seem to appreciate the value in something that is clear and simple, even though it is usually far harder to produce something clear and simple than it is to produce something "new" and complex.
I also wonder whether, in an AI future and the access it appears to provide to collective group-think on a massive scale, we will value textbooks more for pointing out the errors in the conclusions that AI might be trawling from the decisions or whether we will come to accept that the law is whatever AI says that it is.
I do love reading your writing about perverse incentives in academia, Katy. The law too is a vast collection of incentives, many of which have a perverse operation!