Contributed by:

The quest is to understand how best we as educators can teach children and young people is ongoing. We understand a great deal thanks to the work of many scholars and academics who have spent decades studying effective teaching and, in no small way, also to teachers themselves who have been honing their craft over their careers.

1.
Dr Jenni Ingram, Prof. Pam Sammons and Dr Ariel Lindorff

Oxford University

Observing effective

teaching: a review

of the literature

Oxford University

Observing effective

teaching: a review

of the literature

2.

3.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

Observing effective

teaching: a review

of the literature

Education Development Trust Highbridge House, 16–18 Duke Street, Reading, Berkshire RG1 4RU

T +44 (0) 118 902 1000 E enquiries@educationdevelopmenttrust.com W www.educationdevelopmenttrust.com

1

Observing effective

teaching: a review

of the literature

Education Development Trust Highbridge House, 16–18 Duke Street, Reading, Berkshire RG1 4RU

T +44 (0) 118 902 1000 E enquiries@educationdevelopmenttrust.com W www.educationdevelopmenttrust.com

1

4.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

© COPYRIGHT EDUCATION DEVELOPMENT TRUST 2018. THE VIEWS AND OPINIONS EXPRESSED IN THIS PUBLICATION

ARE THOSE OF THE AUTHORS AND DO NOT NECESSARILY REPRESENT THE VIEWS OF EDUCATION DEVELOPMENT TRUST.

ISBN 978-1-909437-97-5

© COPYRIGHT EDUCATION DEVELOPMENT TRUST 2018. THE VIEWS AND OPINIONS EXPRESSED IN THIS PUBLICATION

ARE THOSE OF THE AUTHORS AND DO NOT NECESSARILY REPRESENT THE VIEWS OF EDUCATION DEVELOPMENT TRUST.

ISBN 978-1-909437-97-5

5.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

Contents Welcome to Education Development Trust 4

About the authors 5

Acknowledgements 5

Foreword 7

Chapter 1: Introduction 8

Different frameworks for classroom 9

observations

Breadth of focus 10

Reliability 10

Validity 11

Observation frameworks 12

Chapter 2: Tools for the observation 14

of effective teaching

The International System for Teacher 15

Observation and Feedback

The Quality of Teaching framework 18

The Mathematics Education Traditions 21

of Europe project

Chapter 3: The use of frameworks 24

for research in the UK

Conclusion: Comparing the ISTOF, QoT 26

and METE frameworks

Chapter 4: Alternative frameworks 28

The Knowledge Quartet 29

Mathematical Quality of Instruction 30

Watson's framework 32

Conclusion 32

Chapter 5: Observing to improve teaching 34

Observing effective mathematics teaching 35

Observing to develop teaching 36

Chapter 6: Conclusion 38

References 41

3

Contents Welcome to Education Development Trust 4

About the authors 5

Acknowledgements 5

Foreword 7

Chapter 1: Introduction 8

Different frameworks for classroom 9

observations

Breadth of focus 10

Reliability 10

Validity 11

Observation frameworks 12

Chapter 2: Tools for the observation 14

of effective teaching

The International System for Teacher 15

Observation and Feedback

The Quality of Teaching framework 18

The Mathematics Education Traditions 21

of Europe project

Chapter 3: The use of frameworks 24

for research in the UK

Conclusion: Comparing the ISTOF, QoT 26

and METE frameworks

Chapter 4: Alternative frameworks 28

The Knowledge Quartet 29

Mathematical Quality of Instruction 30

Watson's framework 32

Conclusion 32

Chapter 5: Observing to improve teaching 34

Observing effective mathematics teaching 35

Observing to develop teaching 36

Chapter 6: Conclusion 38

References 41

3

6.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

Welcome to Education Development Trust

At Education Development Trust, we have been improving education around the

world for 50 years. We design and implement improvement programmes for school

systems, and provide consultancy services deploying specialists internationally.

Our work is informed by our continually refreshed body of research which focuses

on the bright spots in education, from education authorities as diverse as those in

Vietnam, Kenya, England, New York and Dubai.

Bringing about real change that alters the aspects of a national system that,

for many reasons, aren’t working so well at the time, requires knowledge and

ability to design and implement changes to any of the levers that can impede

great educational outcomes. So the ability to affect policy, practices, pedagogy,

behaviour, funding, attitudes and more is a prerequisite for a company that can

truly claim to transform lives through improving education.

As highly informed agents of change operating in low- to high-income countries

with their varying internal contexts, we not only design but also show and enable,

so when working with us, everyone involved, from policymakers to school

leaders and teachers, is able to apply their new knowledge to drive sustainable

system reform.

Our expert knowledge, programme design and implementation expertise is also

deployed in delivering Ofsted-rated outstanding careers services in England, and

in owning and managing a family of independent schools.

We are a not-for-profit and we are driven by our values of integrity, accountability,

excellence and collaboration.

Welcome to Education Development Trust

At Education Development Trust, we have been improving education around the

world for 50 years. We design and implement improvement programmes for school

systems, and provide consultancy services deploying specialists internationally.

Our work is informed by our continually refreshed body of research which focuses

on the bright spots in education, from education authorities as diverse as those in

Vietnam, Kenya, England, New York and Dubai.

Bringing about real change that alters the aspects of a national system that,

for many reasons, aren’t working so well at the time, requires knowledge and

ability to design and implement changes to any of the levers that can impede

great educational outcomes. So the ability to affect policy, practices, pedagogy,

behaviour, funding, attitudes and more is a prerequisite for a company that can

truly claim to transform lives through improving education.

As highly informed agents of change operating in low- to high-income countries

with their varying internal contexts, we not only design but also show and enable,

so when working with us, everyone involved, from policymakers to school

leaders and teachers, is able to apply their new knowledge to drive sustainable

system reform.

Our expert knowledge, programme design and implementation expertise is also

deployed in delivering Ofsted-rated outstanding careers services in England, and

in owning and managing a family of independent schools.

We are a not-for-profit and we are driven by our values of integrity, accountability,

excellence and collaboration.

7.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

About the authors

Pam Sammons is a Professor of Education at the Department of Education,

University of Oxford and a Senior Research Fellow at Jesus College, Oxford.

Previously she was a professor at the School of Education, University of

Nottingham (2004-2009) and a professor at the Institute of Education University of

London (1993- 2004) where she directed the International School Effectiveness &

Improvement Centre (ISEIC) 1999-2004. Her research over more than 30 years has

focused on school effectiveness and improvement, school leadership, teaching

effectiveness and professional development, and promoting equity and inclusion

in education.

Jenni Ingram is an Associate Professor of Mathematics Education at the

Department of Education, University of Oxford and Vice Principal of Linacre

College, Oxford. Previously she was an Assistant Professor at the University

of Warwick and a secondary mathematics teacher in inner-city schools in the

Midlands. Her research focuses on mathematics education and mathematics

teacher education, particularly the role of language in the teaching and learning of

mathematics and the using videos in teacher professional development.

Ariel Lindorff is a Research Fellow in the Oxford University Department of

Education. She completed her DPhil at the University of Oxford, and was previously

a secondary mathematics teacher in inner-city settings in the USA. Her research

focuses on educational effectiveness, improvement and equity, often involving

advanced quantitative methods and/or mixed methods.

Funding for this literature review was provided by the Department for Education

(DfE) as part of the TALIS Video Study. The Organisation for Economic Co-

operation and Development (OECD) led TALIS Video study, is a pioneering,

international study, seeking to improve understanding of which aspects of teaching

are related to pupil learning, and the nature of those relationships. The study

focuses particularly on the teaching of mathematics in secondary schools. Over

approximately one year, more than 750 mathematics teachers in eight countries

(including England) will take part in the study. The study will provide a valuable

opportunity to explore how teachers teach in different countries and contexts.

Education Development Trust, in partnership with Oxford University, manages

England’s involvement in the study on behalf of DfE. Fieldwork is ongoing,

commencing in October 2017 and due to end in July 2018.

5

About the authors

Pam Sammons is a Professor of Education at the Department of Education,

University of Oxford and a Senior Research Fellow at Jesus College, Oxford.

Previously she was a professor at the School of Education, University of

Nottingham (2004-2009) and a professor at the Institute of Education University of

London (1993- 2004) where she directed the International School Effectiveness &

Improvement Centre (ISEIC) 1999-2004. Her research over more than 30 years has

focused on school effectiveness and improvement, school leadership, teaching

effectiveness and professional development, and promoting equity and inclusion

in education.

Jenni Ingram is an Associate Professor of Mathematics Education at the

Department of Education, University of Oxford and Vice Principal of Linacre

College, Oxford. Previously she was an Assistant Professor at the University

of Warwick and a secondary mathematics teacher in inner-city schools in the

Midlands. Her research focuses on mathematics education and mathematics

teacher education, particularly the role of language in the teaching and learning of

mathematics and the using videos in teacher professional development.

Ariel Lindorff is a Research Fellow in the Oxford University Department of

Education. She completed her DPhil at the University of Oxford, and was previously

a secondary mathematics teacher in inner-city settings in the USA. Her research

focuses on educational effectiveness, improvement and equity, often involving

advanced quantitative methods and/or mixed methods.

Funding for this literature review was provided by the Department for Education

(DfE) as part of the TALIS Video Study. The Organisation for Economic Co-

operation and Development (OECD) led TALIS Video study, is a pioneering,

international study, seeking to improve understanding of which aspects of teaching

are related to pupil learning, and the nature of those relationships. The study

focuses particularly on the teaching of mathematics in secondary schools. Over

approximately one year, more than 750 mathematics teachers in eight countries

(including England) will take part in the study. The study will provide a valuable

opportunity to explore how teachers teach in different countries and contexts.

Education Development Trust, in partnership with Oxford University, manages

England’s involvement in the study on behalf of DfE. Fieldwork is ongoing,

commencing in October 2017 and due to end in July 2018.

5

8.

9.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE

The quest to understand how best we as educators can teach children and young

people is ongoing. We understand a great deal thanks to the work of many scholars

and academics who have spent decades studying effective teaching and, in no

small way, also to teachers themselves who have been honing their craft over their

careers. Despite this, there is still so much we do not know. A recent Organisation

for Economic Co-operation and Development (OECD) paper described the

‘Supporting teachers has become a top priority across the globe for

the improvement of the quality of our education systems. This renewed

commitment to the teaching profession is based on evidence that teachers are

what makes the greatest difference to learning outside students’ backgrounds,

and that the quality of our school systems is only as good as the quality of

our teachers. A better understanding of what teaching looks like and which

approaches are most effective is not a trivial matter. It is critical, for teaching is

at the heart of a teacher’s role and of the education process.’ 1

Observation is a tool used in teaching and in educational research, albeit in

very different ways in these different disciplines. This review focuses on its use

in research and the study of effective teaching. Observation has tremendous

power to further our understanding about teaching and learning. The education

community is becoming increasingly engaged in the use of video. Simple,

commonplace technologies are helping. This review came about as a result of our

own engagement in an international study, the pilot OECD Teaching and Learning

International Survey (TALIS) Video Study, which uses video to capture and analyse

approaches to teaching mathematics. The OECD report goes on to state:

‘[…] we lack strong evidence about how teaching influences student outcomes

and little is based on actual observation of classroom processes […]. Video-

based research methods now offer an opportunity to understand what teaching

looks like across the globe and, in turn, to enable teachers to learn from their

peers […].’ 2

The tools and approaches that educational researchers employ to understand

teaching through observation are a bit of a mystery to many people, particularly

non-researchers. This review is a clear and concise descriptive summary of the

most tried and tested tools, frameworks and approaches that researchers use to

analyse ‘observed’ teaching. It is an interesting read for anyone involved in the

conduct of observation studies linked to effective teaching, particularly where

mathematics is an area of focus.

Anna Riggall (PhD)

Head of Research,

Education Development Trust

OECD (2018: 2) 2 Ibid.

7

The quest to understand how best we as educators can teach children and young

people is ongoing. We understand a great deal thanks to the work of many scholars

and academics who have spent decades studying effective teaching and, in no

small way, also to teachers themselves who have been honing their craft over their

careers. Despite this, there is still so much we do not know. A recent Organisation

for Economic Co-operation and Development (OECD) paper described the

‘Supporting teachers has become a top priority across the globe for

the improvement of the quality of our education systems. This renewed

commitment to the teaching profession is based on evidence that teachers are

what makes the greatest difference to learning outside students’ backgrounds,

and that the quality of our school systems is only as good as the quality of

our teachers. A better understanding of what teaching looks like and which

approaches are most effective is not a trivial matter. It is critical, for teaching is

at the heart of a teacher’s role and of the education process.’ 1

Observation is a tool used in teaching and in educational research, albeit in

very different ways in these different disciplines. This review focuses on its use

in research and the study of effective teaching. Observation has tremendous

power to further our understanding about teaching and learning. The education

community is becoming increasingly engaged in the use of video. Simple,

commonplace technologies are helping. This review came about as a result of our

own engagement in an international study, the pilot OECD Teaching and Learning

International Survey (TALIS) Video Study, which uses video to capture and analyse

approaches to teaching mathematics. The OECD report goes on to state:

‘[…] we lack strong evidence about how teaching influences student outcomes

and little is based on actual observation of classroom processes […]. Video-

based research methods now offer an opportunity to understand what teaching

looks like across the globe and, in turn, to enable teachers to learn from their

peers […].’ 2

The tools and approaches that educational researchers employ to understand

teaching through observation are a bit of a mystery to many people, particularly

non-researchers. This review is a clear and concise descriptive summary of the

most tried and tested tools, frameworks and approaches that researchers use to

analyse ‘observed’ teaching. It is an interesting read for anyone involved in the

conduct of observation studies linked to effective teaching, particularly where

mathematics is an area of focus.

Anna Riggall (PhD)

Head of Research,

Education Development Trust

OECD (2018: 2) 2 Ibid.

7

10.
Chapter 1

11.
CHAPTER 1: INTRODUCTION

This review examines a range of lesson

observation frameworks designed for and

used in the observation of teaching in

mathematics. This includes frameworks

specifically designed for international

comparisons of teaching practices and

teacher effectiveness, as well as those used

for teaching development.

Classroom observations can be used in a variety of ways, but they are primarily for

the evaluation of teaching, for making comparisons, for professional development

There is a complex

or for a combination of these. There is a complex relationship between teaching

relationship

and learning, and observations of lessons is just one way of examining this

between teaching

relationship. As learning cannot be observed directly, observations usually focus

and learning, and

on identifying particular features of the teaching behaviour and the student

observations of

response. Links are then made with other sources of information, such as student

lessons is just one

attainment and progress measures, student ratings of teaching or lesson artefacts,

way of examining

such as examples of students’ work completed in the lesson.

this relationship

Different frameworks for classroom observations

There is a wide range of observation frameworks available and each is designed

to serve a different purpose. The frameworks included in this review are offered as

examples of this diversity and to exemplify the issues around the design, usability,

validity and reliability of classroom observations. The focus is on frameworks that

have focused specifically on the teaching of mathematics but the review also

includes two generic frameworks used specifically for international comparisons of

the quality and effectiveness of teaching. These two generic frameworks also serve

to illustrate the influence of subject matter on the classroom observation process.

The review excludes frameworks to evaluate the success of particular interventions

or policy initiatives, as these are typically designed to focus on a narrower range

of classroom behaviours and are driven by underlying theories of what ‘good’

teaching might be.

Two key considerations in the design of lesson observation frameworks are

the purpose of the observation and who will be conducting it. For systematic

observations whose purpose is to identify differences between groups of teachers,

for example international comparisons, both validity and reliability are key factors.

9

This review examines a range of lesson

observation frameworks designed for and

used in the observation of teaching in

mathematics. This includes frameworks

specifically designed for international

comparisons of teaching practices and

teacher effectiveness, as well as those used

for teaching development.

Classroom observations can be used in a variety of ways, but they are primarily for

the evaluation of teaching, for making comparisons, for professional development

There is a complex

or for a combination of these. There is a complex relationship between teaching

relationship

and learning, and observations of lessons is just one way of examining this

between teaching

relationship. As learning cannot be observed directly, observations usually focus

and learning, and

on identifying particular features of the teaching behaviour and the student

observations of

response. Links are then made with other sources of information, such as student

lessons is just one

attainment and progress measures, student ratings of teaching or lesson artefacts,

way of examining

such as examples of students’ work completed in the lesson.

this relationship

Different frameworks for classroom observations

There is a wide range of observation frameworks available and each is designed

to serve a different purpose. The frameworks included in this review are offered as

examples of this diversity and to exemplify the issues around the design, usability,

validity and reliability of classroom observations. The focus is on frameworks that

have focused specifically on the teaching of mathematics but the review also

includes two generic frameworks used specifically for international comparisons of

the quality and effectiveness of teaching. These two generic frameworks also serve

to illustrate the influence of subject matter on the classroom observation process.

The review excludes frameworks to evaluate the success of particular interventions

or policy initiatives, as these are typically designed to focus on a narrower range

of classroom behaviours and are driven by underlying theories of what ‘good’

teaching might be.

Two key considerations in the design of lesson observation frameworks are

the purpose of the observation and who will be conducting it. For systematic

observations whose purpose is to identify differences between groups of teachers,

for example international comparisons, both validity and reliability are key factors.

9

12.
CHAPTER 1: INTRODUCTION

Similarly, if the purpose of an observation is to make a judgement about the quality

of teaching then reliability is also a key factor. However, as Coe et al.3 point out,

classroom observations that identify teachers as ‘above average’ or ‘below average’

are accurate only about 60% of the time. In value-added measures of teacher

effectiveness (based on changes in student attainment) the issue of measurement

error is acknowledged as important. The use of ‘confidence intervals’ 4 seeks to

address the concept of statistical uncertainty. Measurement error also applies to

observation instruments and this is why inter-rater reliability measures are required

and observations over more than one lesson are desirable (see Hill, Charalambous

and Kraft 5 for an in-depth discussion of the issue of reliability when observing for

teacher quality).

Breadth of focus

Another key consideration in the design of lesson observation frameworks is the

breadth of focus. Too broad a focus and the framework becomes impractical. The

complexity of the classroom means it is unrealistic to try and observe everything

in a lesson. Decisions therefore need to be made about what to observe, and

these decisions are often framed by the purpose of the schedule or the underlying

theoretical basis of what ‘good’ or ‘effective’ teaching might be. There is also the

The Teaching

potential problem of reducing teaching to a checklist of observable practices,

and Learning

as these cannot take into account the decision-making process behind these

International

behaviours, which are particularly relevant when considering the role of teachers’

Survey (TALIS)

pedagogic subject knowledge. 6 These decisions may be an essential part of what

Video Study 2018

makes the practices effective and be of particular relevance to mathematics

is not designed

teaching, since teachers’ subject knowledge has been found to be more strongly

to judge teachers

linked to variations in student outcomes than it is in some other subjects.7

but rather to be a

source of evidence

based research on

typical patterns

Reliability refers to the extent to which a judgement about a lesson could be of teaching and

replicated. A wide range of factors can affect the reliability of a classroom their associations

observation framework score, including the topic being taught, the individual with a range of

teacher and the observer. However, the extent to which the reliability of an outcomes

observation framework matters depends on the use of the framework. The issue

of reliability is particularly important if observations are used in high-stakes

judgements about individual teachers, such as in relation to teacher promotion or

pay increases, rather than as a research instrument to establish variation in practice

and associations with student outcomes. The Teaching and Learning International

Survey (TALIS) Video Study 2018 is not designed to judge teachers but rather to

be a source of evidence based research on typical patterns of teaching and their

associations with a range of outcomes, including student attainment.

To improve the reliability of lesson observation frameworks, the focus tends to

be on observable behaviours that are ‘low inference’ – that is, the observer is

recording whether something occurs or not without making any judgement about

Coe et al. (2014) 4 95% confidence intervals mean that, if we were to measure teacher effectiveness 100 times, 95 of the intervals generated would contain the true, unobservable, measure

of teacher effectiveness 5 Hill, Charalambous and Kraft (2012) 6 Hewitt (2005) 7 see Hill et al. (2008)

Similarly, if the purpose of an observation is to make a judgement about the quality

of teaching then reliability is also a key factor. However, as Coe et al.3 point out,

classroom observations that identify teachers as ‘above average’ or ‘below average’

are accurate only about 60% of the time. In value-added measures of teacher

effectiveness (based on changes in student attainment) the issue of measurement

error is acknowledged as important. The use of ‘confidence intervals’ 4 seeks to

address the concept of statistical uncertainty. Measurement error also applies to

observation instruments and this is why inter-rater reliability measures are required

and observations over more than one lesson are desirable (see Hill, Charalambous

and Kraft 5 for an in-depth discussion of the issue of reliability when observing for

teacher quality).

Breadth of focus

Another key consideration in the design of lesson observation frameworks is the

breadth of focus. Too broad a focus and the framework becomes impractical. The

complexity of the classroom means it is unrealistic to try and observe everything

in a lesson. Decisions therefore need to be made about what to observe, and

these decisions are often framed by the purpose of the schedule or the underlying

theoretical basis of what ‘good’ or ‘effective’ teaching might be. There is also the

The Teaching

potential problem of reducing teaching to a checklist of observable practices,

and Learning

as these cannot take into account the decision-making process behind these

International

behaviours, which are particularly relevant when considering the role of teachers’

Survey (TALIS)

pedagogic subject knowledge. 6 These decisions may be an essential part of what

Video Study 2018

makes the practices effective and be of particular relevance to mathematics

is not designed

teaching, since teachers’ subject knowledge has been found to be more strongly

to judge teachers

linked to variations in student outcomes than it is in some other subjects.7

but rather to be a

source of evidence

based research on

typical patterns

Reliability refers to the extent to which a judgement about a lesson could be of teaching and

replicated. A wide range of factors can affect the reliability of a classroom their associations

observation framework score, including the topic being taught, the individual with a range of

teacher and the observer. However, the extent to which the reliability of an outcomes

observation framework matters depends on the use of the framework. The issue

of reliability is particularly important if observations are used in high-stakes

judgements about individual teachers, such as in relation to teacher promotion or

pay increases, rather than as a research instrument to establish variation in practice

and associations with student outcomes. The Teaching and Learning International

Survey (TALIS) Video Study 2018 is not designed to judge teachers but rather to

be a source of evidence based research on typical patterns of teaching and their

associations with a range of outcomes, including student attainment.

To improve the reliability of lesson observation frameworks, the focus tends to

be on observable behaviours that are ‘low inference’ – that is, the observer is

recording whether something occurs or not without making any judgement about

Coe et al. (2014) 4 95% confidence intervals mean that, if we were to measure teacher effectiveness 100 times, 95 of the intervals generated would contain the true, unobservable, measure

of teacher effectiveness 5 Hill, Charalambous and Kraft (2012) 6 Hewitt (2005) 7 see Hill et al. (2008)

13.
CHAPTER 1: INTRODUCTION

whether the behaviour is ‘good’ or not. Examples including counting the number

of questions asked or the amount of time spent on different activities. ‘High-

inference’ items require the observer to make judgements about what they are

observing, such as on the clarity of an explanation or the sequencing of the subject

One key way

One key way of improving the reliability of lesson observation scores is to observe of improving

more than one lesson with a specific teacher, covering a range of topics and the reliability

classes. This can help in identifying which variations in practice can be attributed of lesson

to the topic taught or the class being taught. Another issue with observing single observation

lessons and using these to evaluate teachers is that there is a risk that lessons are scores is to

performances and not representative of an individual’s practice in general or over observe more

time. Hill et al. 8 recommend that at least three lessons be observed by at least two than one lesson

different observers to enhance the reliability of observations made. with a specific

teacher, covering

Another common way of improving reliability is for multiple observers to observe

a range of topics

each lesson. Videoing lessons enables the observation of lessons from multiple

and classes

perspectives and with multiple observers. However, videos do not necessarily

capture the full range of what was going on in the lesson, with a single camera

frequently focusing only on the teacher. This can mean a loss of information

around the interactions between students during the lesson, as well as some

interactions between the teacher and the students. Nevertheless, the use of high-

quality frameworks and trained observers and pooling the findings of observations

of multiple lessons by a range of observers can all help improve the reliability of

the observation process.9

These issues of reliability are less of a concern when observation is being used as

a professional development tool. In these situations, it is the feedback that follows

the lesson observation that matters more.10 Teachers often judge observation

feedback to be most useful when it is a subject specialist who can offer advice on

how to improve the lesson who conducts it.11 Evans, Jones and Dawson12 found

that the usefulness of feedback was dependent on whether the observer was a

mathematics specialist or not, and that these judgements were based on the advice

observers offered on how to improve a lesson. Mathematics specialists offered

substantially more suggestions for improvement, with around half of these relating

specifically to the subject-centred aspects of the lesson. In addition, teachers have

identified peer observation as less threatening and as offering a basis for mutual

learning and support to improve practice.13

Validity is a particularly challenging issue in the development of lesson observation

frameworks, especially given the wide range of purposes for which observations

are used. The validity of a framework relates to the extent to which it is measuring

what it is intended to measure. Many mathematics-specific schedules focus on the

observation of teacher subject knowledge and these frameworks can be correlated

with written assessments of this same subject knowledge, although in practice

these correlations are not strong. For example, the Mathematical Quality of

Hill, Charalambous and Kraft (2012) 9 Strong, Gargani and Hacifazlioglu (2011) 10

Coe et al. (2014) 11

Wragg et al. (2002) 12

Evans, Jones and Dawson (2014) 13

Muijs and Reynolds (2005)

11

whether the behaviour is ‘good’ or not. Examples including counting the number

of questions asked or the amount of time spent on different activities. ‘High-

inference’ items require the observer to make judgements about what they are

observing, such as on the clarity of an explanation or the sequencing of the subject

One key way

One key way of improving the reliability of lesson observation scores is to observe of improving

more than one lesson with a specific teacher, covering a range of topics and the reliability

classes. This can help in identifying which variations in practice can be attributed of lesson

to the topic taught or the class being taught. Another issue with observing single observation

lessons and using these to evaluate teachers is that there is a risk that lessons are scores is to

performances and not representative of an individual’s practice in general or over observe more

time. Hill et al. 8 recommend that at least three lessons be observed by at least two than one lesson

different observers to enhance the reliability of observations made. with a specific

teacher, covering

Another common way of improving reliability is for multiple observers to observe

a range of topics

each lesson. Videoing lessons enables the observation of lessons from multiple

and classes

perspectives and with multiple observers. However, videos do not necessarily

capture the full range of what was going on in the lesson, with a single camera

frequently focusing only on the teacher. This can mean a loss of information

around the interactions between students during the lesson, as well as some

interactions between the teacher and the students. Nevertheless, the use of high-

quality frameworks and trained observers and pooling the findings of observations

of multiple lessons by a range of observers can all help improve the reliability of

the observation process.9

These issues of reliability are less of a concern when observation is being used as

a professional development tool. In these situations, it is the feedback that follows

the lesson observation that matters more.10 Teachers often judge observation

feedback to be most useful when it is a subject specialist who can offer advice on

how to improve the lesson who conducts it.11 Evans, Jones and Dawson12 found

that the usefulness of feedback was dependent on whether the observer was a

mathematics specialist or not, and that these judgements were based on the advice

observers offered on how to improve a lesson. Mathematics specialists offered

substantially more suggestions for improvement, with around half of these relating

specifically to the subject-centred aspects of the lesson. In addition, teachers have

identified peer observation as less threatening and as offering a basis for mutual

learning and support to improve practice.13

Validity is a particularly challenging issue in the development of lesson observation

frameworks, especially given the wide range of purposes for which observations

are used. The validity of a framework relates to the extent to which it is measuring

what it is intended to measure. Many mathematics-specific schedules focus on the

observation of teacher subject knowledge and these frameworks can be correlated

with written assessments of this same subject knowledge, although in practice

these correlations are not strong. For example, the Mathematical Quality of

Hill, Charalambous and Kraft (2012) 9 Strong, Gargani and Hacifazlioglu (2011) 10

Coe et al. (2014) 11

Wragg et al. (2002) 12

Evans, Jones and Dawson (2014) 13

Muijs and Reynolds (2005)

11

14.
CHAPTER 1: INTRODUCTION

Instruction (MQI) framework is largely based on this relationship between teacher

performances on written assessments of their subject knowledge and observations

of these same teachers using these schedules. However, these processes for

examining validity are dependent on a range of factors that make them difficult

to carry out. To compare the scores you need an existing instrument that was

designed to measure the same constructs and that has also been extensively tested

for its own validity and reliability. These sorts of studies all need larger sample

sizes than are typically associated with observation studies if they are to generate

confidence in the statistical analyses.

Lesson observation frameworks can also be validated through comparison

with other existing frameworks. If the two frameworks are measuring the same

construct, such as the quality of mathematics teaching, then their scores will

correlate. However, it can be difficult to find frameworks that measure the same

constructs. For example, one study found a low correlation between frameworks

specifically focusing on mathematics teaching and more general frameworks.14

This therefore suggests these frameworks are measuring distinct constructs.

Observation frameworks

This review presents frameworks designed specifically for the comparison

of teacher and school effectiveness internationally. The first of these, the

International System for Teacher Observation and Feedback (ISTOF),15 is a general

framework examining teacher effectiveness, with over 20 countries, including the

UK, involved in its development. This particular framework has been also been This review

used widely for studies of teacher effectiveness within the UK. presents three

The second framework, Quality of Teaching (QoT),16 was similarly designed to frameworks

examine primary teaching quality across four countries, including England, and is designed

also used in teacher effectiveness studies within the UK, often alongside the ISTOF. specifically for

the comparison

The final framework was specifically designed for the observation of mathematics of teacher

lessons, as part of the Mathematics Education Traditions of Europe (METE) project,17 and school

which involved five European countries, including England. The project also effectiveness

focused on the teaching of three particular topics to students aged ten to 14. The internationally

existing research using these three frameworks includes measures of reliability and

validity, making them suitable for research into the quality of teaching.

This review also considers three other frameworks, each designed specifically for

the observation of mathematics lessons. The first, the Knowledge Quartet (KQ),18

developed as part of the Subject Knowledge in Mathematics (SKIMA) research

programme run by the University of Cambridge, 19 was designed specifically to

support the development of mathematics teaching in primary schools in the UK,

particularly among student teachers. The basis of this framework is teacher subject

knowledge and how it influences the teaching of mathematics. It is now used

more broadly by researchers and teacher educators, but remains a tool for teacher

professional development.

Kane and Staiger (2012) 15

Kyriakides et al. (2010) 16

Van de Grift (2007) 17

Andrews (2007) 18

Rowland, Huckstep and Thwaites (2005) 19

See http://www.knowledgequartet.org/introduction/

for further information

Instruction (MQI) framework is largely based on this relationship between teacher

performances on written assessments of their subject knowledge and observations

of these same teachers using these schedules. However, these processes for

examining validity are dependent on a range of factors that make them difficult

to carry out. To compare the scores you need an existing instrument that was

designed to measure the same constructs and that has also been extensively tested

for its own validity and reliability. These sorts of studies all need larger sample

sizes than are typically associated with observation studies if they are to generate

confidence in the statistical analyses.

Lesson observation frameworks can also be validated through comparison

with other existing frameworks. If the two frameworks are measuring the same

construct, such as the quality of mathematics teaching, then their scores will

correlate. However, it can be difficult to find frameworks that measure the same

constructs. For example, one study found a low correlation between frameworks

specifically focusing on mathematics teaching and more general frameworks.14

This therefore suggests these frameworks are measuring distinct constructs.

Observation frameworks

This review presents frameworks designed specifically for the comparison

of teacher and school effectiveness internationally. The first of these, the

International System for Teacher Observation and Feedback (ISTOF),15 is a general

framework examining teacher effectiveness, with over 20 countries, including the

UK, involved in its development. This particular framework has been also been This review

used widely for studies of teacher effectiveness within the UK. presents three

The second framework, Quality of Teaching (QoT),16 was similarly designed to frameworks

examine primary teaching quality across four countries, including England, and is designed

also used in teacher effectiveness studies within the UK, often alongside the ISTOF. specifically for

the comparison

The final framework was specifically designed for the observation of mathematics of teacher

lessons, as part of the Mathematics Education Traditions of Europe (METE) project,17 and school

which involved five European countries, including England. The project also effectiveness

focused on the teaching of three particular topics to students aged ten to 14. The internationally

existing research using these three frameworks includes measures of reliability and

validity, making them suitable for research into the quality of teaching.

This review also considers three other frameworks, each designed specifically for

the observation of mathematics lessons. The first, the Knowledge Quartet (KQ),18

developed as part of the Subject Knowledge in Mathematics (SKIMA) research

programme run by the University of Cambridge, 19 was designed specifically to

support the development of mathematics teaching in primary schools in the UK,

particularly among student teachers. The basis of this framework is teacher subject

knowledge and how it influences the teaching of mathematics. It is now used

more broadly by researchers and teacher educators, but remains a tool for teacher

professional development.

Kane and Staiger (2012) 15

Kyriakides et al. (2010) 16

Van de Grift (2007) 17

Andrews (2007) 18

Rowland, Huckstep and Thwaites (2005) 19

See http://www.knowledgequartet.org/introduction/

for further information

15.
CHAPTER 1: INTRODUCTION

A second framework, referred to as the Watson framework, developed in the UK,20

again was designed with a focus on developing mathematics teaching but this time

at the secondary level. This framework starts from the position of identifying what

aspects of mathematics are being made available to students through the teaching,

rather than focusing on teaching characteristics.

The final framework originates from the US and focuses on the evaluation of

mathematics teaching. The Mathematical Quality of Instruction (MQI) framework,21

has been developed over several years by a team at Harvard, led by Heather

Hill, and is now used in a variety of countries to evaluate mathematics teaching.

Similarly to the Watson Framework, the focus is on mathematical content and how

it is made available to students; similarly to KQ, there is a focus on teacher subject

knowledge. However, in contrast with both KQ and the Watson framework, the MQI

was designed to provide scores for individual mathematics teachers on a number

of discrete dimensions of their mathematics teaching. It is also one of the few

mathematics-specific observation frameworks where there has been considerable

research examining both its validity and its reliability, both within the US and in

other cultural contexts, by means of triangulation of evidence from tests of teacher

subject knowledge, observations of practice and ‘value-added’ measures of

student attainment outcomes.22

Watson (2007) 21

See https://cepr.harvard.edu/mqi for further information 22

See, for example, Hill, Rowan and Ball (2005); Kane and Staiger (2012)

13

A second framework, referred to as the Watson framework, developed in the UK,20

again was designed with a focus on developing mathematics teaching but this time

at the secondary level. This framework starts from the position of identifying what

aspects of mathematics are being made available to students through the teaching,

rather than focusing on teaching characteristics.

The final framework originates from the US and focuses on the evaluation of

mathematics teaching. The Mathematical Quality of Instruction (MQI) framework,21

has been developed over several years by a team at Harvard, led by Heather

Hill, and is now used in a variety of countries to evaluate mathematics teaching.

Similarly to the Watson Framework, the focus is on mathematical content and how

it is made available to students; similarly to KQ, there is a focus on teacher subject

knowledge. However, in contrast with both KQ and the Watson framework, the MQI

was designed to provide scores for individual mathematics teachers on a number

of discrete dimensions of their mathematics teaching. It is also one of the few

mathematics-specific observation frameworks where there has been considerable

research examining both its validity and its reliability, both within the US and in

other cultural contexts, by means of triangulation of evidence from tests of teacher

subject knowledge, observations of practice and ‘value-added’ measures of

student attainment outcomes.22

Watson (2007) 21

See https://cepr.harvard.edu/mqi for further information 22

See, for example, Hill, Rowan and Ball (2005); Kane and Staiger (2012)

13

16.
Chapter 2

Tools for the

observation of

effective teaching

Tools for the

observation of

effective teaching

17.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

The International System for Teacher

Observation and Feedback was developed

by researchers working across 20

countries, including the UK, to specifically

explore the effectiveness of teaching

internationally. 23

The International System for Teacher Observation The ISTOF

and Feedback protocol includes

21 indicators

The ISTOF system schedule is based on recording the extent to which the

grouped

observer agrees that a particular item's description has been observed. The

into seven

fact that this framework offers feedback for teachers is based on research and

components of

expert opinion from more than 20 countries, and that it has been used in a

effective teaching

variety of educational contexts, makes it a useful and reliable framework for

observing teaching.

This schedule has been used in effectiveness studies within England,24

combined with the QoT schedule below. The ISTOF has also been used in

England in the evaluation of Teach First.25

The ISTOF protocol includes 21 indicators grouped into seven components

of effective teaching, as table 1, overleaf, shows. There are 45 items in total,

which are rated on a five-point Likert scale, from strongly agree (5) to strongly

disagree (1), with the option of indicating that it was not possible to observe

some features when they were not relevant to or observable in the particular

classroom setting.

Kyriakides et al. (2010) 24

Day et al. (2008); Sammons et al. (2014) 25

Muijs, Chapman and Armstrong (2012)

15

The International System for Teacher

Observation and Feedback was developed

by researchers working across 20

countries, including the UK, to specifically

explore the effectiveness of teaching

internationally. 23

The International System for Teacher Observation The ISTOF

and Feedback protocol includes

21 indicators

The ISTOF system schedule is based on recording the extent to which the

grouped

observer agrees that a particular item's description has been observed. The

into seven

fact that this framework offers feedback for teachers is based on research and

components of

expert opinion from more than 20 countries, and that it has been used in a

effective teaching

variety of educational contexts, makes it a useful and reliable framework for

observing teaching.

This schedule has been used in effectiveness studies within England,24

combined with the QoT schedule below. The ISTOF has also been used in

England in the evaluation of Teach First.25

The ISTOF protocol includes 21 indicators grouped into seven components

of effective teaching, as table 1, overleaf, shows. There are 45 items in total,

which are rated on a five-point Likert scale, from strongly agree (5) to strongly

disagree (1), with the option of indicating that it was not possible to observe

some features when they were not relevant to or observable in the particular

classroom setting.

Kyriakides et al. (2010) 24

Day et al. (2008); Sammons et al. (2014) 25

Muijs, Chapman and Armstrong (2012)

15

18.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 1: ISTOF PROTOCOL 26

Category Indicator Item

Assessment and The teacher gives explicit, • The teacher makes explicitly clear why an answer is correct or not

evaluation detailed and constructive

• The teacher provides his/her feedback on the answers given by the students

feedback

Assessment is aligned with • Assignments given by the teacher are clearly related to what students learned

goals and objectives

• The teacher explains how assignments are aligned to the learning goals of the

lesson

Differentiation and The teacher creates an • Students communicate frequently with one another on task-oriented issues

inclusion environment in which all

• Students actively engage in learning

students are involved

The teacher takes full account • The teacher makes a distinction in the scope of the assignments for different

of student differences groups of students

• The teacher gives additional opportunities for practice to students who need them

Clarity of instruction The teacher shows good • The teacher regularly checks for understanding

communication skills

• The teacher communicates in a clear and understandable manner

There is clear explanation • The teacher clearly explains the purposes of the lesson

of purpose

• The teacher asks students to identify the reasons why specific activities take place

in the lesson

Lessons are well structured • The teacher presents the lesson with a logical flow that moves from simple to more

complex concepts

• The teacher implements the lesson smoothly, moving from one stage to another

with well-managed transition points

Instructional skills The teacher is able to engage • The teacher provides sufficient wait time and response strategies to involve all

students types of students

• The teacher gives assignments that stimulate all students to active involvement

The teacher possesses good • The teacher poses questions that encourage thinking and elicit feedback

questioning skills

• The length of the pause following questions varies according to the difficulty level

of questions (e.g. a question calling for application of abstract principles requires a

longer pause than a factual question)

The teacher uses various • The teacher uses a variety of instructional strategies during the lesson

teaching methods and

• The teacher uses different strategies for different groups of students

strategies

Promoting active The teacher helps students • The teacher invites students to use strategies that can help them solve different

learning and developing develop problem-solving and types of problems

metacognitive skills meta-cognitive strategies

• The teacher invites students to explain the different steps of the problem-solving

strategy they are using

• The teacher explicitly provides instruction in problem-solving strategies

The teacher gives students • The teacher encourages students to ask one another questions and to explain their

opportunities to be active understanding of topics to one another

learners

• The teacher gives students the opportunity to correct their own work

The teacher fosters critical • The teacher motivates the students to think about the advantages and

thinking in students disadvantages of certain approaches

• The teacher asks the students to reflect on the solutions/answers they give to

problems or questions

• The teacher invites the students to give their personal opinion on certain issues

The teacher connects • The teacher systematically uses material and examples from the students’ daily life

material to students’ to illustrate the course content

real-world experiences

• Students are invited to give their own examples

Adapted from Kyriakides et al. (2010)

TABLE 1: ISTOF PROTOCOL 26

Category Indicator Item

Assessment and The teacher gives explicit, • The teacher makes explicitly clear why an answer is correct or not

evaluation detailed and constructive

• The teacher provides his/her feedback on the answers given by the students

feedback

Assessment is aligned with • Assignments given by the teacher are clearly related to what students learned

goals and objectives

• The teacher explains how assignments are aligned to the learning goals of the

lesson

Differentiation and The teacher creates an • Students communicate frequently with one another on task-oriented issues

inclusion environment in which all

• Students actively engage in learning

students are involved

The teacher takes full account • The teacher makes a distinction in the scope of the assignments for different

of student differences groups of students

• The teacher gives additional opportunities for practice to students who need them

Clarity of instruction The teacher shows good • The teacher regularly checks for understanding

communication skills

• The teacher communicates in a clear and understandable manner

There is clear explanation • The teacher clearly explains the purposes of the lesson

of purpose

• The teacher asks students to identify the reasons why specific activities take place

in the lesson

Lessons are well structured • The teacher presents the lesson with a logical flow that moves from simple to more

complex concepts

• The teacher implements the lesson smoothly, moving from one stage to another

with well-managed transition points

Instructional skills The teacher is able to engage • The teacher provides sufficient wait time and response strategies to involve all

students types of students

• The teacher gives assignments that stimulate all students to active involvement

The teacher possesses good • The teacher poses questions that encourage thinking and elicit feedback

questioning skills

• The length of the pause following questions varies according to the difficulty level

of questions (e.g. a question calling for application of abstract principles requires a

longer pause than a factual question)

The teacher uses various • The teacher uses a variety of instructional strategies during the lesson

teaching methods and

• The teacher uses different strategies for different groups of students

strategies

Promoting active The teacher helps students • The teacher invites students to use strategies that can help them solve different

learning and developing develop problem-solving and types of problems

metacognitive skills meta-cognitive strategies

• The teacher invites students to explain the different steps of the problem-solving

strategy they are using

• The teacher explicitly provides instruction in problem-solving strategies

The teacher gives students • The teacher encourages students to ask one another questions and to explain their

opportunities to be active understanding of topics to one another

learners

• The teacher gives students the opportunity to correct their own work

The teacher fosters critical • The teacher motivates the students to think about the advantages and

thinking in students disadvantages of certain approaches

• The teacher asks the students to reflect on the solutions/answers they give to

problems or questions

• The teacher invites the students to give their personal opinion on certain issues

The teacher connects • The teacher systematically uses material and examples from the students’ daily life

material to students’ to illustrate the course content

real-world experiences

• Students are invited to give their own examples

Adapted from Kyriakides et al. (2010)

19.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 1: ISTOF PROTOCOL (CONTINUED) 26

Category Indicator Item

Classroom climate All students are valued • The teacher demonstrates genuine warmth and empathy towards all students in

the classroom

• The teacher shows respect for the students in both his/her behaviour and the use

of language

The teacher initiates active • The teacher creates purposeful activities that engage every student in productive

interaction and participation work

• The teacher’s instruction is interactive (lots of questions and answers)

The teacher interacts with all • The teacher gives turns to and/or involves those students who do not voluntarily

students participate in classroom activities

• The teacher seeks to engage all students in classroom activities

The teacher communicates • The teacher praises students for effort towards realising their potential

high expectations

• The teacher makes clear that all students know that he/she expects their best

efforts in the classroom

Classroom management Learning time is maximised • The teacher starts the lesson on time

• The teacher makes sure students are involved in learning activities until the end of

the lesson

• Actions are taken to minimise disruption

Clear rules are evident • There is clarity about when and how students can get help

• There is clarity about what options are available when the students finish their

assignments

Misbehaviour and disruptions • The teacher corrects misbehaviour with measures that fit the seriousness of the

are effectively dealt with misconduct (e.g. s/he does not overact)

• The teacher deals with misbehaviour and disruptions by referring to the

established rules of the classroom

Adapted from Kyriakides et al. (2010)

17

TABLE 1: ISTOF PROTOCOL (CONTINUED) 26

Category Indicator Item

Classroom climate All students are valued • The teacher demonstrates genuine warmth and empathy towards all students in

the classroom

• The teacher shows respect for the students in both his/her behaviour and the use

of language

The teacher initiates active • The teacher creates purposeful activities that engage every student in productive

interaction and participation work

• The teacher’s instruction is interactive (lots of questions and answers)

The teacher interacts with all • The teacher gives turns to and/or involves those students who do not voluntarily

students participate in classroom activities

• The teacher seeks to engage all students in classroom activities

The teacher communicates • The teacher praises students for effort towards realising their potential

high expectations

• The teacher makes clear that all students know that he/she expects their best

efforts in the classroom

Classroom management Learning time is maximised • The teacher starts the lesson on time

• The teacher makes sure students are involved in learning activities until the end of

the lesson

• Actions are taken to minimise disruption

Clear rules are evident • There is clarity about when and how students can get help

• There is clarity about what options are available when the students finish their

assignments

Misbehaviour and disruptions • The teacher corrects misbehaviour with measures that fit the seriousness of the

are effectively dealt with misconduct (e.g. s/he does not overact)

• The teacher deals with misbehaviour and disruptions by referring to the

established rules of the classroom

Adapted from Kyriakides et al. (2010)

17

20.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

The Quality of Teaching framework The QoT

framework was

The QoT framework was developed by school inspection teams from four

developed by

countries, including England,27 to inspect the quality of teaching across these

school inspection

countries in primary schools. 28 This is a value-based framework with high-

teams from

inference codes requiring the observer to balance the strengths and

four countries,

weaknesses of different features of the classroom practice being observed.

including

The observer awards an overall grade designed to reflect an overall judgement

England, to

of lesson quality. The initial development of this framework included studies

inspect the quality

that examined the reliability, inter-rater reliability and validity of the observation

of teaching across

framework, specifically focusing on the teaching of mathematics in primary

these countries in

schools. Subsequently, the use of the framework has been extended to other

primary schools

curriculum areas and to secondary lessons within the UK.29

The QoT framework requires trained observers to make professional judgements

about the practice being observed. It draws on the professional judgement

systems used by inspectorates in multiple countries, alongside the educational

effectiveness research literature. It has been tested in a number of European

countries, which has shown that the measures are reliable and mostly scalar

equivalent between different countries.30

The framework itself has six quality characteristics, and each item within these

includes examples of ‘good practice’ to improve the reliability of the judgements

observers make, as table 2 shows. When the completing the table during a lesson

observation, observers must:31

• Score each item on a 1–4 scale depending on the balance of strengths and

weaknesses. The observer places a teacher on the scale according to the

following:

1 = predominantly weak;

2 = more weaknesses than strengths;

3 = more strengths than weaknesses;

4 = predominantly strong.

The observer must score 3 only when all good practice examples (if applicable)

are really observed.

• Circle the correct answer:

0 = no, I didn’t observe this;

1 = yes, I have observed this.

27

Also Belgium, France and the Netherlands 28

Van de Grift (2007) 29

See Day et al. (2008); Sammons et al. (2014) 30

Van de Grift (2013) 31

Van de Grift (2007:148–152)

18

The Quality of Teaching framework The QoT

framework was

The QoT framework was developed by school inspection teams from four

developed by

countries, including England,27 to inspect the quality of teaching across these

school inspection

countries in primary schools. 28 This is a value-based framework with high-

teams from

inference codes requiring the observer to balance the strengths and

four countries,

weaknesses of different features of the classroom practice being observed.

including

The observer awards an overall grade designed to reflect an overall judgement

England, to

of lesson quality. The initial development of this framework included studies

inspect the quality

that examined the reliability, inter-rater reliability and validity of the observation

of teaching across

framework, specifically focusing on the teaching of mathematics in primary

these countries in

schools. Subsequently, the use of the framework has been extended to other

primary schools

curriculum areas and to secondary lessons within the UK.29

The QoT framework requires trained observers to make professional judgements

about the practice being observed. It draws on the professional judgement

systems used by inspectorates in multiple countries, alongside the educational

effectiveness research literature. It has been tested in a number of European

countries, which has shown that the measures are reliable and mostly scalar

equivalent between different countries.30

The framework itself has six quality characteristics, and each item within these

includes examples of ‘good practice’ to improve the reliability of the judgements

observers make, as table 2 shows. When the completing the table during a lesson

observation, observers must:31

• Score each item on a 1–4 scale depending on the balance of strengths and

weaknesses. The observer places a teacher on the scale according to the

following:

1 = predominantly weak;

2 = more weaknesses than strengths;

3 = more strengths than weaknesses;

4 = predominantly strong.

The observer must score 3 only when all good practice examples (if applicable)

are really observed.

• Circle the correct answer:

0 = no, I didn’t observe this;

1 = yes, I have observed this.

27

Also Belgium, France and the Netherlands 28

Van de Grift (2007) 29

See Day et al. (2008); Sammons et al. (2014) 30

Van de Grift (2013) 31

Van de Grift (2007:148–152)

18

21.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING 32

Rate Indicators: The teacher… Observed

Efficient classroom • gives a well-structured lesson 1234 • ensures clearly recognisable components in the lessons 01

management (lesson structure)

• ensures the orderly progression 1234 • ensures entering and leaving the classroom take place in an 01

of the lesson orderly manner

• intervenes in a timely and appropriate way to any order

disruptions

• acts as a ‘watchdog’ for agreed codes of behaviour and rules

• uses learning time efficiently 1234 • ensures there is no loss of time at the start, during or at the 01

end of the lesson

• ensures there are no ‘dead’ moments

• ensures the students are not left waiting

• ensures efficient classroom 1234 • makes clear which lesson materials should be used 01

management

• ensures the lesson materials are ready to use

• ensures the lesson materials are adapted to the level and

experience of the students

Safe and stimulating • ensures a relaxed atmosphere 1234 • addresses the children in a positive manner 01

learning climate

• reacts with humour, and stimulates humour

• allows children to make mistakes

• promotes mutual respect 1234 • encourages students to listen to one another 01

• intervenes when students are being laughed at

• takes (cultural) differences and idiosyncrasies into account

• supports the self-confidence of 1234 • feeds back on questions and answers from students in a 01

students positive way

• expresses positive expectations to students about what they

are able to take on

• shows respect for the students in 1234 • allows students to finish speaking 01

behaviour and language use

• listens to what students have to say

• makes no role-confirming remarks

• ensures cohesion 1234 • honours the contributions made by students 01

• ensures solidarity between students

• ensures events are experienced as group events

• stimulates the independence of 1234 • allows students to work independently on another 01

students assignment or to take up an individually selected task after

completing an assignment

• allows students to work with self-correcting materials

• has students working on daily and weekly tasks

• promotes cooperation between 1234 • provides opportunities for students to help one another 01

students

• gives assignments that incite cooperation

• gives students the opportunity to play together or to carry

out assignments together

Clear instruction • clarifies the lesson objectives at 1234 • informs students at the start of the lesson about the aims of 01

the start of the lesson the lesson

• clarifies the aim of the assignment and what the students

will learn from it

• evaluates whether the objectives 1234 • verifies and/or evaluates whether the aims of the lesson 01

have been achieved at the end of have been achieved

the lesson

• checks the students’ achievements

Adapted from Van de Grift (2007: 148–152)

19

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING 32

Rate Indicators: The teacher… Observed

Efficient classroom • gives a well-structured lesson 1234 • ensures clearly recognisable components in the lessons 01

management (lesson structure)

• ensures the orderly progression 1234 • ensures entering and leaving the classroom take place in an 01

of the lesson orderly manner

• intervenes in a timely and appropriate way to any order

disruptions

• acts as a ‘watchdog’ for agreed codes of behaviour and rules

• uses learning time efficiently 1234 • ensures there is no loss of time at the start, during or at the 01

end of the lesson

• ensures there are no ‘dead’ moments

• ensures the students are not left waiting

• ensures efficient classroom 1234 • makes clear which lesson materials should be used 01

management

• ensures the lesson materials are ready to use

• ensures the lesson materials are adapted to the level and

experience of the students

Safe and stimulating • ensures a relaxed atmosphere 1234 • addresses the children in a positive manner 01

learning climate

• reacts with humour, and stimulates humour

• allows children to make mistakes

• promotes mutual respect 1234 • encourages students to listen to one another 01

• intervenes when students are being laughed at

• takes (cultural) differences and idiosyncrasies into account

• supports the self-confidence of 1234 • feeds back on questions and answers from students in a 01

students positive way

• expresses positive expectations to students about what they

are able to take on

• shows respect for the students in 1234 • allows students to finish speaking 01

behaviour and language use

• listens to what students have to say

• makes no role-confirming remarks

• ensures cohesion 1234 • honours the contributions made by students 01

• ensures solidarity between students

• ensures events are experienced as group events

• stimulates the independence of 1234 • allows students to work independently on another 01

students assignment or to take up an individually selected task after

completing an assignment

• allows students to work with self-correcting materials

• has students working on daily and weekly tasks

• promotes cooperation between 1234 • provides opportunities for students to help one another 01

students

• gives assignments that incite cooperation

• gives students the opportunity to play together or to carry

out assignments together

Clear instruction • clarifies the lesson objectives at 1234 • informs students at the start of the lesson about the aims of 01

the start of the lesson the lesson

• clarifies the aim of the assignment and what the students

will learn from it

• evaluates whether the objectives 1234 • verifies and/or evaluates whether the aims of the lesson 01

have been achieved at the end of have been achieved

the lesson

• checks the students’ achievements

Adapted from Van de Grift (2007: 148–152)

19

22.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32

Rate Indicators: The teacher… Observed

Clear instruction • gives clear instructions and 1234 • activates the student’s prior knowledge 01

(continued) explanations

• explains in sequential stages

• asks questions that are understood by the students

• summarises the lesson materials from time to time

• gives clear explanations of 1234 • ensures that every student knows what he/she has to do 01

the learning materials and the

• clearly indicates the materials that can be used as

assignments

learning aids

• gives feedback to students 1234 • checks whether students have understood the lesson 01

materials when he/she is instructing the class

• checks whether students are completing the assignments

correctly

• gives feedback on the way students arrive at their answers

• gives feedback on the social functioning involved in the

completion of the tasks (group work)

• involves all students in the lesson 1234 • gives assignments that stimulate students into active 01

involvement

• poses questions that initiate reflection

• ensures students listen carefully and keep on working

• waits sufficiently long to allow students to reflect after

posing a question

• gives the opportunity to respond to students who don’t put

their hands up

• makes use of teaching methods 1234 • makes use of conversational forms and discussion forms 01

that activate the students

• provides graduated exercises

• permits working in groups/corners

• makes use of information and communication technology

Adaption of • adapts the instruction to the 1234 • allows students who need less instruction to commence 01

teaching relevant differences between with the work

students

• gives extra instruction to small groups or individual students

• does not direct himself exclusively to the middle bracket

• adapts the assignments and 1234 • makes a distinction in the scope of the assignments between 01

processing to the relevant individual students

differences between students

• does not give all students the same time to complete the

assignment

• allows some students to make use of auxiliary materials

Teaching learning • ensures the teaching materials are 1234 • teaches students solution strategies or search and reference 01

strategies oriented towards transfer strategies

• teaches students the use of organisation resources

• promotes the conscious use of what has been learned in

other (different) areas of learning

• stimulates the use of control 1234 • gives attention to estimatory calculation/anticipatory 01

activities reading

• has solutions related to the context

• stimulates the use of alternative solutions

• provides interactive instruction 1234 • facilitates mutual interaction between students… ensures 01

and activities interaction between pupils and the teacher

Adapted from Van de Grift (2007: 148–152)

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32

Rate Indicators: The teacher… Observed

Clear instruction • gives clear instructions and 1234 • activates the student’s prior knowledge 01

(continued) explanations

• explains in sequential stages

• asks questions that are understood by the students

• summarises the lesson materials from time to time

• gives clear explanations of 1234 • ensures that every student knows what he/she has to do 01

the learning materials and the

• clearly indicates the materials that can be used as

assignments

learning aids

• gives feedback to students 1234 • checks whether students have understood the lesson 01

materials when he/she is instructing the class

• checks whether students are completing the assignments

correctly

• gives feedback on the way students arrive at their answers

• gives feedback on the social functioning involved in the

completion of the tasks (group work)

• involves all students in the lesson 1234 • gives assignments that stimulate students into active 01

involvement

• poses questions that initiate reflection

• ensures students listen carefully and keep on working

• waits sufficiently long to allow students to reflect after

posing a question

• gives the opportunity to respond to students who don’t put

their hands up

• makes use of teaching methods 1234 • makes use of conversational forms and discussion forms 01

that activate the students

• provides graduated exercises

• permits working in groups/corners

• makes use of information and communication technology

Adaption of • adapts the instruction to the 1234 • allows students who need less instruction to commence 01

teaching relevant differences between with the work

students

• gives extra instruction to small groups or individual students

• does not direct himself exclusively to the middle bracket

• adapts the assignments and 1234 • makes a distinction in the scope of the assignments between 01

processing to the relevant individual students

differences between students

• does not give all students the same time to complete the

assignment

• allows some students to make use of auxiliary materials

Teaching learning • ensures the teaching materials are 1234 • teaches students solution strategies or search and reference 01

strategies oriented towards transfer strategies

• teaches students the use of organisation resources

• promotes the conscious use of what has been learned in

other (different) areas of learning

• stimulates the use of control 1234 • gives attention to estimatory calculation/anticipatory 01

activities reading

• has solutions related to the context

• stimulates the use of alternative solutions

• provides interactive instruction 1234 • facilitates mutual interaction between students… ensures 01

and activities interaction between pupils and the teacher

Adapted from Van de Grift (2007: 148–152)

23.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32

Rate Indicators: The teacher… Observed

Involvement • ensures there is good individual 1234 • ensures pupils actively listen to the instructions 01

of pupils involvement by the pupils

• ensures pupils take part in learning/group discussions

• ensures pupils work on the assignment in a concentrated,

task-focused way

Final judgement The overall quality of teaching is 1234

judged as:

The Mathematics Education Traditions of Europe project

The METE observation framework developed out of a study comparing

mathematics teaching in five European countries: England, Finland, Flanders

Belgium, Hungary and Spain. The schedule was developed through live

observations and then used video recordings of lessons for the main analyses. The

lessons focused on the teaching of specific topics with students aged ten to 14:

percentages, polygons and linear equations. The main focus of the study was on

how mathematics teachers structured students’ opportunities for learning.33 The

framework consists of three broad categories, each containing several foci, which

were designed to be easily applied across all observers in all countries involved.

Each category was designed to be low inference and to address observable

behaviours in the lessons. The first category refers to the mathematical foci or

observable learning outcomes, as table 3 shows.

TABLE 3: METE MATHEMATICAL FOCI 34

Mathematical foci Description – the teacher is seen to emphasise or encourage:

Conceptual the conceptual development of his or her students

Derivational the process of developing new mathematical entities from existing knowledge

Structural the links or connections between different mathematical entities, concepts, properties, etc.

Procedural the acquisition of skills, procedures, techniques or algorithms

Efficiency pupils’ understanding or acquisition of processes or techniques that develop flexibility, elegance or critical

comparison of working

Problem solving pupils’ engagement with the solution of non-trivial or non-routine tasks

Reasoning pupils’ development and articulation of justification and argumentation

The second category for observation focuses on the contexts in which the teachers

posed the tasks. It has two dimensions: (1) whether the context was related to the real

world or not and (2) whether the data or information used was genuine or invented

by the teacher. In this way, the assessment of mathematics classroom activity can be

carried out using a two-dimensional grid, as shown in table 4, overleaf.

Adapted from Van de Grift (2007: 148–152) 33

Andrews (2009) 34

Adapted from Andrews (2007: 501)

21

TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32

Rate Indicators: The teacher… Observed

Involvement • ensures there is good individual 1234 • ensures pupils actively listen to the instructions 01

of pupils involvement by the pupils

• ensures pupils take part in learning/group discussions

• ensures pupils work on the assignment in a concentrated,

task-focused way

Final judgement The overall quality of teaching is 1234

judged as:

The Mathematics Education Traditions of Europe project

The METE observation framework developed out of a study comparing

mathematics teaching in five European countries: England, Finland, Flanders

Belgium, Hungary and Spain. The schedule was developed through live

observations and then used video recordings of lessons for the main analyses. The

lessons focused on the teaching of specific topics with students aged ten to 14:

percentages, polygons and linear equations. The main focus of the study was on

how mathematics teachers structured students’ opportunities for learning.33 The

framework consists of three broad categories, each containing several foci, which

were designed to be easily applied across all observers in all countries involved.

Each category was designed to be low inference and to address observable

behaviours in the lessons. The first category refers to the mathematical foci or

observable learning outcomes, as table 3 shows.

TABLE 3: METE MATHEMATICAL FOCI 34

Mathematical foci Description – the teacher is seen to emphasise or encourage:

Conceptual the conceptual development of his or her students

Derivational the process of developing new mathematical entities from existing knowledge

Structural the links or connections between different mathematical entities, concepts, properties, etc.

Procedural the acquisition of skills, procedures, techniques or algorithms

Efficiency pupils’ understanding or acquisition of processes or techniques that develop flexibility, elegance or critical

comparison of working

Problem solving pupils’ engagement with the solution of non-trivial or non-routine tasks

Reasoning pupils’ development and articulation of justification and argumentation

The second category for observation focuses on the contexts in which the teachers

posed the tasks. It has two dimensions: (1) whether the context was related to the real

world or not and (2) whether the data or information used was genuine or invented

by the teacher. In this way, the assessment of mathematics classroom activity can be

carried out using a two-dimensional grid, as shown in table 4, overleaf.

Adapted from Van de Grift (2007: 148–152) 33

Andrews (2009) 34

Adapted from Andrews (2007: 501)

21

24.

25.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING

TABLE 4: METE CONTEXT 35

Dimensions Example

The task is explicitly related to • The task of calculating the cost of decorating a hypothetical room is related to the real world but is located in a fantasy

the real world and based on of data – the dimensions of the room, the costs of paper, for example.

data or entities invented by the

• Revising the cost of a pair of hypothetical trousers after a sale reduction.

The task is explicitly not related • An invitation to solve the equation x2–3x+1=0 is not based in the real world and the data or entity – the equation itself

to the real world and based on – is not the product of a student’s own activity.

data or entities invented by the

• Many text-based questions or exercises would fall into this category.

The task is explicitly related to • Testing statistical hypotheses derived from real data collected by students.

the real world and based on

• Calculating the cost of manufacturing a desk by measuring the desk. It is the act of measurement, which creates

genuine data or entities

genuine data, that feeds back into the real world, as it addresses the cost of making the desks.

The task is explicitly not related • Exploring the minimum value of a quadratic expression of the student’s choice has no explicit relation to the real world,

to the real world and based on but the data – the choice of the individual student – is real.

genuine data or entities

• An invitation to students to measure the length of their desks for no other purpose than to practise the skills of

measurement. The task is not explicitly related to the real world because it does not feed back into it, but it is located in

real-world, genuine data. In this scenario, the real world provides a background context for the task.

The final category concerns teacher strategies, or ‘mathematical didactics’, that

might be used to facilitate students' learning of mathematics, as table 5 shows.

These categories were used both when the teacher was working with a class as a

whole and when the students were working individually or as a small group.

TABLE 5: METE TEACHING STRATEGY 36

Teaching strategy Description

Activating prior knowledge Focuses students' attention on mathematical content covered earlier in their careers via a period of revision as

preparation for activities to follow.

Exercising prior knowledge Focuses students’ attention on mathematical content covered earlier in their careers via a period of revision unrelated to

any activities that follow.

Explaining Explains an idea or solution. This could include demonstration, explicitly telling or the pedagogic modelling of higher-

level thinking. In such instances, the teacher is the informer with little or no student input.

Sharing Engages students in the sharing of ideas, solutions or answers. This could include class discussions, where the teacher's

role is one of manager rather than informer.

Exploring Engages students in an activity, not teacher directed, from which a new mathematical idea is intended to emerge.

This activity could be an investigation or a sequence of structured problems, but in all cases students are expected to

articulate their findings.

Coaching The teacher explicitly offers hints, prompts or feedback to facilitate students’ understanding of or ability to perform tasks

or to correct misunderstandings.

Assessing or evaluating Assesses or evaluates students’ responses to determine the overall attainment of the class.

Motivating Addresses students’ attitudes, beliefs or emotional responses towards mathematics.

Questioning The teacher explicitly uses a sequence of questions, perhaps Socratic, so as to lead students to construct new

mathematical ideas or clarify or refine existing ones.

Differentiation Attempts to treat students differently in terms of the kind of activities performed, materials provided and/or the expected

outcome to make instruction optimally adapted to the students’ characteristics and needs.

Adapted from Andrews (2007) 36

Adapted from Andrews (2007: 503)

23

TABLE 4: METE CONTEXT 35

Dimensions Example

The task is explicitly related to • The task of calculating the cost of decorating a hypothetical room is related to the real world but is located in a fantasy

the real world and based on of data – the dimensions of the room, the costs of paper, for example.

data or entities invented by the

• Revising the cost of a pair of hypothetical trousers after a sale reduction.

The task is explicitly not related • An invitation to solve the equation x2–3x+1=0 is not based in the real world and the data or entity – the equation itself

to the real world and based on – is not the product of a student’s own activity.

data or entities invented by the

• Many text-based questions or exercises would fall into this category.

The task is explicitly related to • Testing statistical hypotheses derived from real data collected by students.

the real world and based on

• Calculating the cost of manufacturing a desk by measuring the desk. It is the act of measurement, which creates

genuine data or entities

genuine data, that feeds back into the real world, as it addresses the cost of making the desks.

The task is explicitly not related • Exploring the minimum value of a quadratic expression of the student’s choice has no explicit relation to the real world,

to the real world and based on but the data – the choice of the individual student – is real.

genuine data or entities

• An invitation to students to measure the length of their desks for no other purpose than to practise the skills of

measurement. The task is not explicitly related to the real world because it does not feed back into it, but it is located in

real-world, genuine data. In this scenario, the real world provides a background context for the task.

The final category concerns teacher strategies, or ‘mathematical didactics’, that

might be used to facilitate students' learning of mathematics, as table 5 shows.

These categories were used both when the teacher was working with a class as a

whole and when the students were working individually or as a small group.

TABLE 5: METE TEACHING STRATEGY 36

Teaching strategy Description

Activating prior knowledge Focuses students' attention on mathematical content covered earlier in their careers via a period of revision as

preparation for activities to follow.

Exercising prior knowledge Focuses students’ attention on mathematical content covered earlier in their careers via a period of revision unrelated to

any activities that follow.

Explaining Explains an idea or solution. This could include demonstration, explicitly telling or the pedagogic modelling of higher-

level thinking. In such instances, the teacher is the informer with little or no student input.

Sharing Engages students in the sharing of ideas, solutions or answers. This could include class discussions, where the teacher's

role is one of manager rather than informer.

Exploring Engages students in an activity, not teacher directed, from which a new mathematical idea is intended to emerge.

This activity could be an investigation or a sequence of structured problems, but in all cases students are expected to

articulate their findings.

Coaching The teacher explicitly offers hints, prompts or feedback to facilitate students’ understanding of or ability to perform tasks

or to correct misunderstandings.

Assessing or evaluating Assesses or evaluates students’ responses to determine the overall attainment of the class.

Motivating Addresses students’ attitudes, beliefs or emotional responses towards mathematics.

Questioning The teacher explicitly uses a sequence of questions, perhaps Socratic, so as to lead students to construct new

mathematical ideas or clarify or refine existing ones.

Differentiation Attempts to treat students differently in terms of the kind of activities performed, materials provided and/or the expected

outcome to make instruction optimally adapted to the students’ characteristics and needs.

Adapted from Andrews (2007) 36

Adapted from Andrews (2007: 503)

23

26.
Chapter 3

The use of

frameworks for

research in the UK

The use of

frameworks for

research in the UK

27.
CHAPTER 3: THE USE OF FRAMEWORKS FOR RESEARCH IN THE UK

The ISTOF, QoT and METE frameworks,

have all been used by researchers in the

UK to examine teaching, and specifically

mathematics teaching, through

observations of teaching practice.

These studies often use the frameworks in conjunction with other sources of data,

such as teacher interviews and student questionnaires, in order to gain a fuller

The effective

description of more effective teaching. The findings from these studies relate

classroom practice

closely to the existing literature on effective teaching and learning.

study used the

The effective classroom practice study used the ISTOF and QoT,37 alongside teacher ISTOF and QoT,

questionnaires and interviews, school leader interviews and pupil questionnaires alongside teacher

and interviews, to establish a multidimensional picture of effective classroom questionnaires

practice. The frameworks specifically identified several core characteristics of and interviews,

more effective teaching. Specifically, the ISTOF identified clear and coherent school leader

lessons with a supportive learning climate; engaging students with assignments interviews

and activities; positive classroom management; purposive learning; and quality and pupil

questioning and feedback for students. The more effective teachers also questionnaires

scored very highly on the QoT characteristics of a supportive lesson climate; and interviews

proactive lesson management; well-organised lessons with clear objectives; and

environmental and teacher support.

The inspiring teachers study took a mixed-methods approach to characterising

inspiring teachers,38 again using both the ISTOF and QoT, combined with

qualitative observations, teacher and school leader interviews, and pupil

questionnaires. The sample included 17 teachers representing primary and

secondary schools. The comparison between the qualitative observations and the

use of the two quantitative frameworks, the ISTOF and QoT, revealed that inspiring

teachers also showed strongly the characteristics of more effective teaching. In

particular, the teachers identified as inspiring scored particularly highly on the

ISTOF in relation to creating a positive classroom climate, classroom management

and clarity of instruction. Similarly, these teachers also scored highly on the QoT

components related to a safe and orderly school climate, effective classroom

layout, clear instruction and effective classroom organisation. As such, while

defining inspiring teachers relies more on ideas, such as student engagement

and enjoyment, than on effectiveness studies that focus on student academic

outcomes, the findings of this study conclude that inspiring teachers are first

and foremost highly effective teachers.

The ISTOF has also been used by Muijs, Chapman and Armstrong39 to explore the

effectiveness of Teach First teachers (an alternative teacher certification programme

in England). Similarly to the studies above, this framework was used in conjunction

See Day et al. (2008): Kington et al. (2012) 38

Sammons et al. (2014) 39

Muijs, Chapman and Armstrong (2012)

25

The ISTOF, QoT and METE frameworks,

have all been used by researchers in the

UK to examine teaching, and specifically

mathematics teaching, through

observations of teaching practice.

These studies often use the frameworks in conjunction with other sources of data,

such as teacher interviews and student questionnaires, in order to gain a fuller

The effective

description of more effective teaching. The findings from these studies relate

classroom practice

closely to the existing literature on effective teaching and learning.

study used the

The effective classroom practice study used the ISTOF and QoT,37 alongside teacher ISTOF and QoT,

questionnaires and interviews, school leader interviews and pupil questionnaires alongside teacher

and interviews, to establish a multidimensional picture of effective classroom questionnaires

practice. The frameworks specifically identified several core characteristics of and interviews,

more effective teaching. Specifically, the ISTOF identified clear and coherent school leader

lessons with a supportive learning climate; engaging students with assignments interviews

and activities; positive classroom management; purposive learning; and quality and pupil

questioning and feedback for students. The more effective teachers also questionnaires

scored very highly on the QoT characteristics of a supportive lesson climate; and interviews

proactive lesson management; well-organised lessons with clear objectives; and

environmental and teacher support.

The inspiring teachers study took a mixed-methods approach to characterising

inspiring teachers,38 again using both the ISTOF and QoT, combined with

qualitative observations, teacher and school leader interviews, and pupil

questionnaires. The sample included 17 teachers representing primary and

secondary schools. The comparison between the qualitative observations and the

use of the two quantitative frameworks, the ISTOF and QoT, revealed that inspiring

teachers also showed strongly the characteristics of more effective teaching. In

particular, the teachers identified as inspiring scored particularly highly on the

ISTOF in relation to creating a positive classroom climate, classroom management

and clarity of instruction. Similarly, these teachers also scored highly on the QoT

components related to a safe and orderly school climate, effective classroom

layout, clear instruction and effective classroom organisation. As such, while

defining inspiring teachers relies more on ideas, such as student engagement

and enjoyment, than on effectiveness studies that focus on student academic

outcomes, the findings of this study conclude that inspiring teachers are first

and foremost highly effective teachers.

The ISTOF has also been used by Muijs, Chapman and Armstrong39 to explore the

effectiveness of Teach First teachers (an alternative teacher certification programme

in England). Similarly to the studies above, this framework was used in conjunction

See Day et al. (2008): Kington et al. (2012) 38

Sammons et al. (2014) 39

Muijs, Chapman and Armstrong (2012)

25

28.
CHAPTER 3: THE USE OF FRAMEWORKS FOR RESEARCH IN THE UK

with interviews with teachers and school leaders, and teacher questionnaires. The

Teach First teachers demonstrated high levels of the behaviours in the framework

that are considered indicators of more effective teachers and they also scored

similarly to those teachers observed during the design of the ISTOF framework. 40

Conclusion: Comparing the ISTOF, QoT and METE frameworks

Whilst the ISTOF and QoT share some similarities in terms of their components

and measures, they are both conceptually and practically very different measures

of teaching behaviours. Both schedules show sufficient reliable test results for

use in the two studies above, but the ISTOF framework scores higher on inter-

rater reliability and reliability than the QoT framework. The correlation between

teachers’ overall scores on the ISTOF and the QoT were strong, positive and

statistically significant. Both frameworks provide an overall measure of effective

practice but also distinguish different features of practice that can be used when The METE

giving feedback to the teachers involved. Similarly, combining these frameworks framework

with field notes can contribute to the usefulness of feedback, as they can provide has been used

more detail on student characteristics and prior learning. to explore

The studies that have used these frameworks in combination suggest that there is similarities

an overall concept of teacher effectiveness, but also that there are differentiations and differences

within this. Consequently, more effective teachers would show both strengths and between

weakness in particular aspects of their practice that might vary over time, with teachers, both

different topics and with different students. The frameworks themselves identify in England and

broad descriptions of more effective practice, but both the effective classroom internationally

practice study and the Inspiring Teachers Study found considerable variation in

the ways that these broad categories of more effective practice were enacted by

the teachers.

The METE framework has been used to explore similarities and differences

between teachers, both in England and internationally. It has not been used to

compare the effectiveness of particular teaching behaviours but rather what

similarities and differences across groups of teachers or across a particular topic,

such as linear equations, can tell us about the learning of mathematics. Similarly to

the Inspiring Teachers Study, whilst some similarities across the broad categories

was observed, there were noticeable differences in the ways that these categories

were observed in teachers’ classroom practice. 41

Kyriakides et al. (2010) 41

Andrews, 2009.

with interviews with teachers and school leaders, and teacher questionnaires. The

Teach First teachers demonstrated high levels of the behaviours in the framework

that are considered indicators of more effective teachers and they also scored

similarly to those teachers observed during the design of the ISTOF framework. 40

Conclusion: Comparing the ISTOF, QoT and METE frameworks

Whilst the ISTOF and QoT share some similarities in terms of their components

and measures, they are both conceptually and practically very different measures

of teaching behaviours. Both schedules show sufficient reliable test results for

use in the two studies above, but the ISTOF framework scores higher on inter-

rater reliability and reliability than the QoT framework. The correlation between

teachers’ overall scores on the ISTOF and the QoT were strong, positive and

statistically significant. Both frameworks provide an overall measure of effective

practice but also distinguish different features of practice that can be used when The METE

giving feedback to the teachers involved. Similarly, combining these frameworks framework

with field notes can contribute to the usefulness of feedback, as they can provide has been used

more detail on student characteristics and prior learning. to explore

The studies that have used these frameworks in combination suggest that there is similarities

an overall concept of teacher effectiveness, but also that there are differentiations and differences

within this. Consequently, more effective teachers would show both strengths and between

weakness in particular aspects of their practice that might vary over time, with teachers, both

different topics and with different students. The frameworks themselves identify in England and

broad descriptions of more effective practice, but both the effective classroom internationally

practice study and the Inspiring Teachers Study found considerable variation in

the ways that these broad categories of more effective practice were enacted by

the teachers.

The METE framework has been used to explore similarities and differences

between teachers, both in England and internationally. It has not been used to

compare the effectiveness of particular teaching behaviours but rather what

similarities and differences across groups of teachers or across a particular topic,

such as linear equations, can tell us about the learning of mathematics. Similarly to

the Inspiring Teachers Study, whilst some similarities across the broad categories

was observed, there were noticeable differences in the ways that these categories

were observed in teachers’ classroom practice. 41

Kyriakides et al. (2010) 41

Andrews, 2009.

29.

30.
Chapter 4

31.
CHAPTER 4: ALTERNATIVE FRAMEWORKS

The three frameworks that follow have

largely been used in different ways from the

three frameworks already discussed.

For example, the Mathematical Quality of Instruction (MQI) framework has been

used widely to evaluate the subject knowledge of mathematics teachers, and the

The validity

number of countries for which it has been validated continues to grow, though

and reliability

there is no study at present showing its validity within the UK. The Knowledge

of the MQI has

Quartet (KQ) has also been used extensively, but as a professional development

been established

tool rather than as a measure of effectiveness or subject knowledge. This contrast

in several

mimics the purposes for which these two frameworks were designed: both were

studies through

initially designed for the observation of primary mathematics lessons but their use

comparison

has extended to secondary mathematics. The Watson framework was specifically

with written

designed for the observation of secondary mathematics classrooms and again has

assessments of

been used predominantly as a professional development tool. The validity and

mathematics

reliability of the MQI has been established in several studies42 through comparison

teachers’ subject

with written assessments of mathematics teachers’ subject knowledge. However,

knowledge

these validity and reliability are not appropriate measures of the usefulness to

professional development. This instead relies on how useful teachers and teacher

educators have found the frameworks in developing their practice.

The Knowledge Quartet

KQ was developed as a framework to support ‘productive discussion of

mathematics content knowledge between teacher educators, trainees and

teacher-mentors’. 43 It was designed as a framework both for lesson observation

and for mathematics teaching development. The focus is on mathematics

subject knowledge and it is designed to develop both mathematics teaching and

mathematics teacher knowledge.

This framework was initially designed through working with primary student

teachers and their university tutors and mentors, and through the analysis of

videos of teaching on teaching practice, but it is now widely used for professional

development purposes at all levels of education.

As table 6 outlines, there are four aspects to KQ. The first category – foundation

knowledge – underpins the other categories, as it focuses on the knowledge and

beliefs of the teacher, with the other categories focusing on the application of that

knowledge in teaching. These categories are not mutually exclusive and episodes

within a lesson can be understood in terms of more than one of them.

For example, ‘a contingent response to a pupil’s suggestion might helpfully

connect with ideas considered earlier’. 44

See, for example, Hill et al. (2008) 43

Rowland, Huckstep and Thwaites (2005: 256) 44

Ibid. (259)

29

The three frameworks that follow have

largely been used in different ways from the

three frameworks already discussed.

For example, the Mathematical Quality of Instruction (MQI) framework has been

used widely to evaluate the subject knowledge of mathematics teachers, and the

The validity

number of countries for which it has been validated continues to grow, though

and reliability

there is no study at present showing its validity within the UK. The Knowledge

of the MQI has

Quartet (KQ) has also been used extensively, but as a professional development

been established

tool rather than as a measure of effectiveness or subject knowledge. This contrast

in several

mimics the purposes for which these two frameworks were designed: both were

studies through

initially designed for the observation of primary mathematics lessons but their use

comparison

has extended to secondary mathematics. The Watson framework was specifically

with written

designed for the observation of secondary mathematics classrooms and again has

assessments of

been used predominantly as a professional development tool. The validity and

mathematics

reliability of the MQI has been established in several studies42 through comparison

teachers’ subject

with written assessments of mathematics teachers’ subject knowledge. However,

knowledge

these validity and reliability are not appropriate measures of the usefulness to

professional development. This instead relies on how useful teachers and teacher

educators have found the frameworks in developing their practice.

The Knowledge Quartet

KQ was developed as a framework to support ‘productive discussion of

mathematics content knowledge between teacher educators, trainees and

teacher-mentors’. 43 It was designed as a framework both for lesson observation

and for mathematics teaching development. The focus is on mathematics

subject knowledge and it is designed to develop both mathematics teaching and

mathematics teacher knowledge.

This framework was initially designed through working with primary student

teachers and their university tutors and mentors, and through the analysis of

videos of teaching on teaching practice, but it is now widely used for professional

development purposes at all levels of education.

As table 6 outlines, there are four aspects to KQ. The first category – foundation

knowledge – underpins the other categories, as it focuses on the knowledge and

beliefs of the teacher, with the other categories focusing on the application of that

knowledge in teaching. These categories are not mutually exclusive and episodes

within a lesson can be understood in terms of more than one of them.

For example, ‘a contingent response to a pupil’s suggestion might helpfully

connect with ideas considered earlier’. 44

See, for example, Hill et al. (2008) 43

Rowland, Huckstep and Thwaites (2005: 256) 44

Ibid. (259)

29

32.
CHAPTER 4: ALTERNATIVE FRAMEWORKS

TABLE 6: THE KNOWLEDGE QUARTET 45

Category Description

Foundation Propositional knowledge and beliefs concerning:

• the meanings and descriptions of relevant mathematical concepts, and the relationships that exist between them;

• the different factors that research has shown to be significant in the teaching and learning of mathematics;

• the ontological status of mathematics and the purposes of teaching it.

Contributory codes: awareness of purpose; identifying errors; overt subject knowledge; theoretical underpinning of

pedagogy; use of terminology; use of textbook; reliance on procedures.

Transformation Knowledge-in-action revealed in deliberation and the choices made in planning and teaching.

The teacher transforms and presents his or her own meanings and descriptions in ways designed to enable students to

learn. These could include the use of powerful analogies, illustrations, explanations and demonstrations.

The choice of examples made by the teacher is especially visible:

• for optimal acquisition of mathematical concepts, procedures or essential vocabulary;

• for confronting common misconceptions;

• for the justification (by generic example) or refutation (by counter-example) of mathematical ideas.

Contributory codes: choice of representation; teacher demonstration; choice of examples.

Connection Knowledge-in-action revealed in deliberation and choice in planning and teaching.

Within a single lesson, or across several lessons, the teacher unifies the subject matter and draws out coherence with

respect to:

• connections between different meanings and descriptions of particular concepts or between alternative ways of

representing concepts and conducting procedures;

• the relative complexity and cognitive demands of mathematical concepts and procedures, by attention to sequencing of

the content.

Contributory codes: making connections between procedures; making connections between concepts; anticipation of

complexity; decisions about sequencing; recognition of conceptual appropriateness

Contingency Knowledge-in-interaction revealed through the ability of the teacher to ‘think on his/her feet’ and respond

appropriately to the contributions made by students during a teaching episode.

This could be seen in the teacher’s willingness to deviate from his/her own agenda, when to develop a student’s

unanticipated contribution:

• might be of special benefit to that pupil; or

• might suggest a particularly fruitful avenue of enquiry for others.

Contributory codes: responding to children’s ideas; use of opportunities; deviation from agenda

Mathematical Quality of Instruction

The MQI was developed by Heather Hill and colleagues at the University of

Michigan and Harvard University to reliably measure several dimensions of the

work teachers do with students around mathematical content. The MQI is based on

a theory of instruction, existing literature on effective instruction in mathematics

and an analysis of nearly 250 videotapes of US teachers and teaching. This means

that the design is flexible enough to consider the variety of mathematics teaching

that occurs in classrooms. The MQI is based on the premise that the mathematical

work that occurs in classrooms is distinct from generic features of teaching, such

as classroom climate.

Adapted from Rowland, Huckstep and Thwaites (2005: 265–266)

TABLE 6: THE KNOWLEDGE QUARTET 45

Category Description

Foundation Propositional knowledge and beliefs concerning:

• the meanings and descriptions of relevant mathematical concepts, and the relationships that exist between them;

• the different factors that research has shown to be significant in the teaching and learning of mathematics;

• the ontological status of mathematics and the purposes of teaching it.

Contributory codes: awareness of purpose; identifying errors; overt subject knowledge; theoretical underpinning of

pedagogy; use of terminology; use of textbook; reliance on procedures.

Transformation Knowledge-in-action revealed in deliberation and the choices made in planning and teaching.

The teacher transforms and presents his or her own meanings and descriptions in ways designed to enable students to

learn. These could include the use of powerful analogies, illustrations, explanations and demonstrations.

The choice of examples made by the teacher is especially visible:

• for optimal acquisition of mathematical concepts, procedures or essential vocabulary;

• for confronting common misconceptions;

• for the justification (by generic example) or refutation (by counter-example) of mathematical ideas.

Contributory codes: choice of representation; teacher demonstration; choice of examples.

Connection Knowledge-in-action revealed in deliberation and choice in planning and teaching.

Within a single lesson, or across several lessons, the teacher unifies the subject matter and draws out coherence with

respect to:

• connections between different meanings and descriptions of particular concepts or between alternative ways of

representing concepts and conducting procedures;

• the relative complexity and cognitive demands of mathematical concepts and procedures, by attention to sequencing of

the content.

Contributory codes: making connections between procedures; making connections between concepts; anticipation of

complexity; decisions about sequencing; recognition of conceptual appropriateness

Contingency Knowledge-in-interaction revealed through the ability of the teacher to ‘think on his/her feet’ and respond

appropriately to the contributions made by students during a teaching episode.

This could be seen in the teacher’s willingness to deviate from his/her own agenda, when to develop a student’s

unanticipated contribution:

• might be of special benefit to that pupil; or

• might suggest a particularly fruitful avenue of enquiry for others.

Contributory codes: responding to children’s ideas; use of opportunities; deviation from agenda

Mathematical Quality of Instruction

The MQI was developed by Heather Hill and colleagues at the University of

Michigan and Harvard University to reliably measure several dimensions of the

work teachers do with students around mathematical content. The MQI is based on

a theory of instruction, existing literature on effective instruction in mathematics

and an analysis of nearly 250 videotapes of US teachers and teaching. This means

that the design is flexible enough to consider the variety of mathematics teaching

that occurs in classrooms. The MQI is based on the premise that the mathematical

work that occurs in classrooms is distinct from generic features of teaching, such

as classroom climate.

Adapted from Rowland, Huckstep and Thwaites (2005: 265–266)

33.
CHAPTER 4: ALTERNATIVE FRAMEWORKS

The MQI uses the three key relationships widely used in mathematics education

research, often referred to as ‘the didactic triangle’. 46 These are the relationships:

between the teacher and the mathematics; between the teacher and the students;

and between the students and the mathematics, as illustrated in table 7. The

framework provides separate teacher scores for five different dimensions,

which can each be used to assess these relationships. These dimensions are

the richness of the mathematics; errors and imprecision; working with students

and mathematics; student participation in meaning making and reasoning; and

connections between classroom work and mathematics.

The framework uses video recordings of lessons. Each recorded lesson is then

divided into roughly equal length (e.g. 5 or 7.5 minute) segments for scoring by two

independent raters. A score is given for each of these five MQI dimensions and the

raters also each give the whole lesson an overall MQI score.

TABLE 7: MQI CONSTRUCTS 47

Construct Description

Teacher–content relationship Richness of the mathematics

Richness includes two pieces:

(1) attention to the meaning of mathematical facts and procedures and

(2) engagement with mathematical practices and language.

Meaning making includes explanations of mathematical ideas and drawing connections among different mathematical

ideas or different representations of the same idea. Mathematical practices are represented by multiple solution methods,

where more credit is given for comparisons of solution methods for ease or efficiency; by developing mathematical

generalisations from examples; and by the fluent and precise use of mathematical language.

Errors and imprecision

This captures whether the teacher makes major errors indicating gaps in mathematical knowledge; whether the teacher

distorts content through unclear articulation of concepts; and/or whether there is a lack of clarity in the presentation of

content or the launching of tasks.

Teacher–student relationship Working with students and mathematics

This investigates whether the teacher accurately interprets and responds to students’ mathematical ideas. It also looks at

whether the teacher can correct student errors thoroughly, with attention to the specific misunderstandings that led to

the errors.

Student–content relationship Student participation in meaning making and reasoning

This captures the ways in which students engage with mathematical content, specifically whether students ask questions

and reason about mathematics; whether students provide mathematical explanations independently or in response to

the teacher’s questions; and/or the cognitive requirements of specific tasks, such as whether students are asked to find

patterns, draw connections or explain and/or justify their conclusions.

Connections between classroom work and mathematics

This explores whether classroom work has a mathematical point, or whether the bulk of instructional time is spent on

activities that do not specifically develop mathematical ideas, for example cutting and pasting or non-productive uses of

time, including transitions or discipline.

Chevallard (1985); Brousseau (1997) 47

Adapted from National Center for Teacher Effectiveness (2012)

31

The MQI uses the three key relationships widely used in mathematics education

research, often referred to as ‘the didactic triangle’. 46 These are the relationships:

between the teacher and the mathematics; between the teacher and the students;

and between the students and the mathematics, as illustrated in table 7. The

framework provides separate teacher scores for five different dimensions,

which can each be used to assess these relationships. These dimensions are

the richness of the mathematics; errors and imprecision; working with students

and mathematics; student participation in meaning making and reasoning; and

connections between classroom work and mathematics.

The framework uses video recordings of lessons. Each recorded lesson is then

divided into roughly equal length (e.g. 5 or 7.5 minute) segments for scoring by two

independent raters. A score is given for each of these five MQI dimensions and the

raters also each give the whole lesson an overall MQI score.

TABLE 7: MQI CONSTRUCTS 47

Construct Description

Teacher–content relationship Richness of the mathematics

Richness includes two pieces:

(1) attention to the meaning of mathematical facts and procedures and

(2) engagement with mathematical practices and language.

Meaning making includes explanations of mathematical ideas and drawing connections among different mathematical

ideas or different representations of the same idea. Mathematical practices are represented by multiple solution methods,

where more credit is given for comparisons of solution methods for ease or efficiency; by developing mathematical

generalisations from examples; and by the fluent and precise use of mathematical language.

Errors and imprecision

This captures whether the teacher makes major errors indicating gaps in mathematical knowledge; whether the teacher

distorts content through unclear articulation of concepts; and/or whether there is a lack of clarity in the presentation of

content or the launching of tasks.

Teacher–student relationship Working with students and mathematics

This investigates whether the teacher accurately interprets and responds to students’ mathematical ideas. It also looks at

whether the teacher can correct student errors thoroughly, with attention to the specific misunderstandings that led to

the errors.

Student–content relationship Student participation in meaning making and reasoning

This captures the ways in which students engage with mathematical content, specifically whether students ask questions

and reason about mathematics; whether students provide mathematical explanations independently or in response to

the teacher’s questions; and/or the cognitive requirements of specific tasks, such as whether students are asked to find

patterns, draw connections or explain and/or justify their conclusions.

Connections between classroom work and mathematics

This explores whether classroom work has a mathematical point, or whether the bulk of instructional time is spent on

activities that do not specifically develop mathematical ideas, for example cutting and pasting or non-productive uses of

time, including transitions or discipline.

Chevallard (1985); Brousseau (1997) 47

Adapted from National Center for Teacher Effectiveness (2012)

31

34.
CHAPTER 4: ALTERNATIVE FRAMEWORKS

Watson’s framework Frameworks have

been designed for

Watson’s framework48 is included as it is different in design from the other

a specific purpose,

frameworks discussed in that it ‘start[s] from mathematics rather than from

such as the subject

teaching’. 49 This framework was designed for use by mathematics teachers, and

knowledge of

particularly student teachers, to improve the teaching of mathematics. Again,

mathematics

the framework focuses on observable teacher behaviours but also includes

teachers or as

considerations of the ‘kinds of shift a learner might be hoped to make during

a professional

mathematical activity’.50 Table 8 presents the ‘dimensions of mathematical

development

pedagogic orientation’, the relevant tasks or prompts that are observable and the

tool, as well as for

shifts required for each dimension.

primary teaching

or secondary

teaching

These three frameworks all focus on the particular features of mathematics

teaching and enable us to observe the mathematical content and its presentation,

as well as some of the more general features focused on in the earlier frameworks.

Each of these frameworks have been designed for a specific purpose, such as the

subject knowledge of mathematics teachers or as a professional development tool,

as well as for primary teaching or secondary teaching. Whilst their use in other

settings or for other purposes has not been validated by research, many teachers

are finding them a useful framework for analysing their own teaching.

Watson (2007) 49

Ibid. (118) 50

Ibid. (120)

Watson’s framework Frameworks have

been designed for

Watson’s framework48 is included as it is different in design from the other

a specific purpose,

frameworks discussed in that it ‘start[s] from mathematics rather than from

such as the subject

teaching’. 49 This framework was designed for use by mathematics teachers, and

knowledge of

particularly student teachers, to improve the teaching of mathematics. Again,

mathematics

the framework focuses on observable teacher behaviours but also includes

teachers or as

considerations of the ‘kinds of shift a learner might be hoped to make during

a professional

mathematical activity’.50 Table 8 presents the ‘dimensions of mathematical

development

pedagogic orientation’, the relevant tasks or prompts that are observable and the

tool, as well as for

shifts required for each dimension.

primary teaching

or secondary

teaching

These three frameworks all focus on the particular features of mathematics

teaching and enable us to observe the mathematical content and its presentation,

as well as some of the more general features focused on in the earlier frameworks.

Each of these frameworks have been designed for a specific purpose, such as the

subject knowledge of mathematics teachers or as a professional development tool,

as well as for primary teaching or secondary teaching. Whilst their use in other

settings or for other purposes has not been validated by research, many teachers

are finding them a useful framework for analysing their own teaching.

Watson (2007) 49

Ibid. (118) 50

Ibid. (120)