Sunday, November 03, 2013

Rabbit Tracks and the Big Game of Education - Jeff Johnson's Understanding of What Makes a Great School for All in Alberta...

“You don’t follow rabbit tracks when you are hunting big game” said Jeff Johnson, Minister of Education in the Government of Alberta at a conference run by the Canadian Education Association. No one knows what the big game is in education in Alberta anymore, but we do know that Jeff Johnson is not the right man to lead the hunt for excellence and school transformation.

In response to a simple question asked by Naresh Bhardwaj (a former teacher) about class size this last week in the legislature he said this:
“when we’re looking at quality of education and the success of the student, the size of the class is not the most important thing to track or to try to affect. Obviously, the engagement of the parent is the most important, but second to that is the quality of teaching.”(my emphasis)

Where to start?

There is a mountain of data dating from the early 1970’s and Michael Rutter’s study of school effectiveness and the work of David Reynolds (both of which I was involved in) looking at the factors which impact school achievement. What we know is that there are a range of factors which impact school performance. While parental engagement is a “nice to have” it is not amongst the most critical in shaping school outcomes.

In terms of what we know after this forty years of very rigorous work is that the five key factors which shape outcomes are: (a) prior educational performance of the student – a student who has done well before is likely to continue to do so; (b) social class and economic status – students from impoverished backgrounds do less well than the students from high income families; (c) school size and culture, both overall size of the school and class size matter for a range of complex reasons; (d) whether the school is urban or rural – which is basically a matter of curriculum choice and ability to attract and retain quality teachers; and (e) whether the school is public, private or Catholic and the level of support it receives for its work from its funding source. This list comes from a comprehensive study of this question undertaken by the Government of Australia (2004).

Class size is a complex question, as Harvey Goldstein and Peter Blatchford  pointed out in the 1990’s (here) – it is not just size, it is what happens in class, who is in class and the degree of student engagement that impacts performance. There is not a simple cause-effect relationship. However, a US analysis and synthesis of almost 200 empirical research studies have shown that, for specific targeted purposes, reducing class size improves academic performance. The targeted purposes are: (a) for primary education; (b) for schools with a high intake of students from poor economic backgrounds; (c) where there is a high level of students in class with special needs; and (d) where the subject being studied is known to be challenging for many students. A key condition of success for small class size is that teachers have been trained to teach small groups.

A meta-analysis of 77 studies published by Smith and Glass in 1978 found that small classes were associated with higher achievement at all grade levels. The major benefits of reduction occurred where the number of students in the class was fewer than 20. They concluded that small classes were superior in terms of students’ reactions, teacher morale, and the quality of the instructional environment.

The key findings to review are those from the detailed analysis of PISA. These show that school climate and culture are far more significant that specific organizational measures. While they are related, culture and context speak more to the engaged environment of all aspects of schooling than any specific organizational measure, such as class size (see here, especially at page 37).

The question of teacher quality, which has preoccupied Jeff Johnson for some time (he favours merit pay for teachers – he made this clear when he worked with Dave Hancock on Inspiring Education) is an important question.  A variety of studies indicate that measures of teacher preparation and certification are by far the strongest correlates of student achievement in reading and mathematics, both before and after controlling for student socio-economic status and that teacher quality and qualifications count (see here). In particular, subject matter knowledge (those who have a degree in mathematics or science are much better at teaching math or science than those who do not), knowledge of teaching for student engagement and skills in the effective use of learning technologies are all seen as key ingredients for student achievement. Surprisingly for some, the key here is knowledge of teaching and learning processes – it is far more important than knowledge of the subject (Ferguson and Womack, 1993).

Not all teachers are great teachers all of the time. The biggest critics of bad teaching are good teachers. As part of any review there is a need to look at sensible professional practice, including peer-to-peer performance measures and continual performance assessment of teaching, including clear metrics for raising performance year after year. There is also a strong case to introduce a requirement for continuing professional development (CPD) for teachers as a requirement for continuing certification. The important thing is to have a clear, effective system that school Principals can use flexibly but is also fair to staff.

Merit pay for teachers is not a smart idea. A 2004 study in Tennessee showed that it had mixed success in rewarding teachers who increased student achievement. Assignment to career-ladder teachers increased mathematics scores by roughly 3 percentile points but generally had smaller and statistically insignificant effects on reading scores (here). Several meta analysis show that merit pay has little or no impact on student achievement, but does start to change adult behaviour in inappropriate ways. Schools have many educational goals – not only easily tested basic skills in math and reading, but the sciences, history, good citizenship, appreciation of literature, the arts and music, physical fitness, good health habits, and character. In any institution with complex or multiple goals, incentive systems that reward achieving only some of those goals (usually those most easily measured) will inevitably distort that system’s output. Rational agents, responding to incentives, will ensure that resources, time, and attention are redirected to goals being rewarded, and away from those (perhaps equally important but more difficult to measure) not being rewarded.  Thirty years ago, the methodologist Donald T. Campbell framed what he called a ‘law’ of performance measurement:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

Since then, social scientists have documented how simple accountability or incentive systems based on quantitative output indicators have actually harmed the institutions they were designed to improve – not only in education but in business, health care, welfare policy, human capital development, criminal justice, and public administration. Merit pay is a rabbit track.

As for parental involvement or engagement, it is the prior education of parents and their economic status which has an impact on student achievement, not their engagement with the school. Parents who read a lot are more likely to provide an environment in which their children read and parents with a post-secondary education are more likely to be able to provide learner supports to their children than parents who did not complete their high school education. But this is more about socio-economic status and income levels than about engagement. According to the PISA data – consistent over time - about
50 per cent or more of differences between schools are jointly explained by the school climate and student characteristics and the school context. Parental engagement does not appear to be a significant factor.

So, I suggest to Naresh Bhardwaj MLA that he pay attention to the evidence and not much attention to Jeff Johnson, especially if Naresh Bhardwaj is seeking to make informed decisions on the basis of evidence. As for Jeff Johnson, well he should go out and shoot some rabbits – he seems to be tracking them.


Ferguson, P., & Womack, S .T. (1993). The impact of subject matter and education
coursework on teaching performance. Journal of Teacher Education, 44 (1), 55-63.

No comments: