The story of an inadvertent controlled experiment in correlations of student performance with teacher evaluation.
Alice recently taught two out of four large lecture sections of an introductory science course on a challenging topic. Alice tries to do something different each semester to improve the course. This time, she pared back extraneous details and focussed the lectures more on understanding over formula-crunching. She integrated breaks where students worked through sample questions in class, and asked them to discuss these questions with their neighbors. She conveyed high expectations.
The other instructor, Bob, followed a traditional lecture style. He felt that some students would succeed, some would fail; there wasn’t much he could do about it, so he would deliver to the students what they wanted: sample exam problems in each lecture.
The four sections of the course were identical in all aspects other than lecture: shared homework, labs, recitations, exams, review sessions. Each lecturer contributed half of the questions on each exam. Alice asked the course administrator to track student performance on the exams by lecture section, to see if the new teaching techniques had any effect. Alice was also curious about the end-of-semester student evaluations. Let’s start with those.
Alice was hammered in the student ratings of teaching effectiveness, by far the lowest ratings of any course she had taught (including this same course in the past). Bob’s ratings were fine – a full two points on a six point scale separated them. Beyond raw hatred and anger, Alice’s students overwhelmingly expressed a single sentiment: It was grossly unfair that Bob’s students received sample exam questions in every lecture and they did not. Alice’s students felt strongly that they were not properly prepared for the exams because they were not taught how to work through each kind of problem.
In reality? Both of Alice’s lecture sections performed better than both of Bob’s sections on every exam. The difference increased as the semester proceeded. Roughly one third of Alice’s students moved up one letter grade increment as a result.
Alice considered possible confounding effects: Do better students tend to choose lectures at a particular time of day? Possibly, but colleagues who have studied the effect said that the time-of-day trend tends to be opposite to what Alice and Bob observed, and there was also no significant time-based sub-trend between the two separate sections that each taught. Did a subpopulation of Alice’s students attend Bob’s lectures? Mixing between the sections would generally suppress differences in performance, not accentuate or reverse them*. Otherwise, this was like a controlled experiment: all course elements beyond lecture were identical. The students punished Alice for lowering their exam scores, when in reality she raised them. How much better might Alice’s students have done had they recognized and engaged with what was working?
Alice’s Dean likes to say that the student evaluations are an imperfect measure of teacher performance. But an imperfect measure is one with a modest positive correlation to the target characteristic, not a negative correlation. Alice notes that the university systematically measures student opinion across course and semester, but does not systematically measure student learning. An institution will reward what it can recognize, but will drift otherwise.
The hatred and anger continue into the followup course in the next semester (with a different co-lecturer), as does the improved performance of Alice’s students. The numerical average of the student evaluations continues to be poor, but the student comments now also contain some lengthy positive statements, several of which note with dismay the attitudes and practices of other students.
What attitudes and practices? For many, this required course is perceived mainly as a barrier against the student gaining admission to his or her desired major: a mechanical hurdle, not an intellectual challenge. Internet discussion groups and commercial websites deliver homework solutions. The “response function” to homework is negative: more challenging homework means more cheating and less total student effort. Local tutoring services deliver plug-and-chug training to muddle through exams. And old sample exams.
No-one wants to think of their own actions as illegitimate. But cheating on homework can be made legitimate, in a student’s mind, if the course can be made illegitimate. The motto of the students’ private Facebook group, joined by a majority of the class: “Seriously, f— this course”.
A psychological challenge to the students’ delegitimization of a course – through an instructor’s high expectations for learning – can elicit a forceful defensive emotional response, to push the illegitimacy back onto the course.
Anti-correlation.
*Assuming that the two sets of lectures initially have similar student populations, whatever subgroup moves between lectures can be matched against an existing equivalent subgroup in the destination lecture that receives the same “dose” of lecture. The combination of both subgroups then contributes nothing to between-lecturer differences in student performance.
