good teaching really does exist
Over at Econlog, I got into a debate with economist Arnold Kling about teacher effectiveness. He takes an extremely skeptical view:
I am afraid that I am skeptical of Rick Hanushek’s claim that the best teachers are really effective and the worst are really ineffective. If that were true, then I think we would observe private schools dramatically outperforming public schools, holding student characteristics constant, and I do not think that is what the data say. Instead, when we see differences, those differences typically do not persist over time.
In education research, intensive efforts are made to find differences caused by teachers or other inputs. This is a worthwhile effort, but whenever studies are published showing such differences, they need to be discounted heavily for the biases induced by various filters in the research and publication process. The likelihood of any strong difference holding up in repeated study is quite low.
Here is my response. I think Arnold is making a huge mistake in interpreting the teacher effectiveness literature. The take home messages are:
- There really are individual teachers who are measurably better than others.
- Easily measured attributes (e.g., credentials, gender, age, seniority, etc) usually do not correlate with effective teaching (i.e., test score improvement).
In other words, Dr. Kling may be a good teacher, but that can’t be explained by his econ PhD or # of years in the classroom. It’s “Kling specific” – a “fixed effect” in statistics jargon.
It’s easy for people to read conclusion #2 and claim that teaching quality is bogus. My interpretation is simpler and makes more sense given that there appear to be some really good teachers.
Teaching is mainly about coaching and connecting with students so that they will endure the material. Teachers also need some internal self-discipline so they will stick to the methods that are known to work (e.g., repetition in arithmetic, or phonics in reading). These two characteristics are really about personality more than credentials or other easily observed characteristics. So that’s why you get the weird results from teacher research – yes for teacher fixed effects, but no effect for covariates such as credentials or school characteristics.
Finally, Arnold is responding to the finding that your school or teacher doesn’t change your entire life. Teacher effects are modest, but educational attainment is a life long process. A good teacher will rarely change the remedial student to college bound material, but it can matter, at least in the short term.
The fact that these are residual “fixed effects” as opposed to factors which correlate with other things is worrysome. Even if teaching were random; I would expect to find a classroom-specific fixed effect, and a corresponding “teacher skill distribution” if I looked at test scores. The fact that these skill parameters don’t seem to show much serial correlation also points to the fact that they may be picking up noise or luck or background factors uncorrelated with actual teaching quality.
But suppose it’s true. Then this makes me worried about the reformers inclination to give bonuses to the best teachers and kick out the worst ones. After all, any distribution of teacher talent should have some *good* and *bad* teachers.
But clearly, we can’t continue a bonus/fire process forever, as the teacher distribution will always have extremes in quality, and in the long run such extreme variances in compensation will draw a skewed teaching pool. But then where should we stop? And if we can’t set a rule determining how much firing is too much, why do we think that we don’t fire enough now?
I don’t doubt that there are good or bad teachers, or that they are relatively uncorrelated with credentials. That should be clear to anyone who has spent time in a classroom. Just that these simple calculations can figure that out and offer straightforward policy advice; any more than simple return calculations are sufficient to identify good mutual fund managers.
Thorfinn
December 29, 2010 at 12:57 am
What bothers me is that so much time is going into defining and labelling teachers as either effective or below average. This time could be better spent encouraging and supporting all teachers, making them feel important rather than scrutinised and assisting them wherever possible. All teachers can improve, but the modern landscape has made teachers particularly self-conscious and unfairly judged.
After all we don’t deliberate nearly as long and hard about the effectiveness of our doctors, policemen and women and dentists.
Michael G.
December 29, 2010 at 1:49 am
following thorfinn, it’s probably worth noting that Kling was responding to McMegan (who was in turn commenting on NYT) about how fixed effects are only mildly reliable and this creates political problems with using them for personnel policy.
another way to put this is to imagine a three-level nested model with 1. student-years within 2. teacher-years within 3. teachers. Kling is pointing out that there’s more variance at 1 than at 2. Fabio is observing that actually there’s a lot of variance at 2 and 3. The NYT and McMegan were pointing out that what we’d ideally like is for very little variance to be at level 2 compared to level 3.
gabrielrossman
December 29, 2010 at 6:00 pm
I am sure that some people are better teachers than others, but I do not believe that test scores are a way to measure this. The recent study that demonstrated that teaching to the test produces lower test scores than other teaching strategies gave me some hope, and maybe test scores could measure math skills, but beyond that I think the relationship breaks down.
Good writing cannot be scored using a mechanistic rubric, as some of the exposes of SAT writing grading sessions show. There are no tests that determine whether people have learned to question the taken-for-granted. There is so much else going on in terms of school-level and student-level characteristics that are hard to control for. And in order to truly determine if a teacher is effective, measures would need to be taken years later–which further complicates matters.
In particular, this seems to be a problem with high-school level teachers, who are often held up as ineffective because they don’t produce sufficient learning gains in students who were turned off to schooling years ago and who are already many years beyond grade level. Rather than focusing on assessing teachers with faulty metrics, resources would be better spent ensuring that small children get started in education on the right foot. And, of course, in elevating teaching careers to the level of selectivity and prestige they get in other countries.
Mikaila
January 2, 2011 at 12:35 am
@Michael G, though all teachers can surely improve, we don’t have any rigorously demonstrated way of making teachers much better. Perhaps we should focus more on how to evaluate doctors and police rather than less on how to evaluate teachers.
Michael Bishop
January 7, 2011 at 12:53 am