Contributed by:
The quest is to understand how best we as educators can teach children and young people is ongoing. We understand a great deal thanks to the work of many scholars and academics who have spent decades studying effective teaching and, in no small way, also to teachers themselves who have been honing their craft over their careers.
1.
Dr Jenni Ingram, Prof. Pam Sammons and Dr Ariel Lindorff
Oxford University
Observing effective
teaching: a review
of the literature
3.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
Observing effective
teaching: a review
of the literature
Education Development Trust Highbridge House, 16–18 Duke Street, Reading, Berkshire RG1 4RU
T +44 (0) 118 902 1000 E [email protected] W www.educationdevelopmenttrust.com
1
4.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
© COPYRIGHT EDUCATION DEVELOPMENT TRUST 2018. THE VIEWS AND OPINIONS EXPRESSED IN THIS PUBLICATION
ARE THOSE OF THE AUTHORS AND DO NOT NECESSARILY REPRESENT THE VIEWS OF EDUCATION DEVELOPMENT TRUST.
ISBN 978-1-909437-97-5
5.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
Contents Welcome to Education Development Trust 4
About the authors 5
Acknowledgements 5
Foreword 7
Chapter 1: Introduction 8
Different frameworks for classroom 9
observations
Breadth of focus 10
Reliability 10
Validity 11
Observation frameworks 12
Chapter 2: Tools for the observation 14
of effective teaching
The International System for Teacher 15
Observation and Feedback
The Quality of Teaching framework 18
The Mathematics Education Traditions 21
of Europe project
Chapter 3: The use of frameworks 24
for research in the UK
Conclusion: Comparing the ISTOF, QoT 26
and METE frameworks
Chapter 4: Alternative frameworks 28
The Knowledge Quartet 29
Mathematical Quality of Instruction 30
Watson's framework 32
Conclusion 32
Chapter 5: Observing to improve teaching 34
Observing effective mathematics teaching 35
Observing to develop teaching 36
Chapter 6: Conclusion 38
References 41
3
6.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
Welcome to Education Development Trust
At Education Development Trust, we have been improving education around the
world for 50 years. We design and implement improvement programmes for school
systems, and provide consultancy services deploying specialists internationally.
Our work is informed by our continually refreshed body of research which focuses
on the bright spots in education, from education authorities as diverse as those in
Vietnam, Kenya, England, New York and Dubai.
Bringing about real change that alters the aspects of a national system that,
for many reasons, aren’t working so well at the time, requires knowledge and
ability to design and implement changes to any of the levers that can impede
great educational outcomes. So the ability to affect policy, practices, pedagogy,
behaviour, funding, attitudes and more is a prerequisite for a company that can
truly claim to transform lives through improving education.
As highly informed agents of change operating in low- to high-income countries
with their varying internal contexts, we not only design but also show and enable,
so when working with us, everyone involved, from policymakers to school
leaders and teachers, is able to apply their new knowledge to drive sustainable
system reform.
Our expert knowledge, programme design and implementation expertise is also
deployed in delivering Ofsted-rated outstanding careers services in England, and
in owning and managing a family of independent schools.
We are a not-for-profit and we are driven by our values of integrity, accountability,
excellence and collaboration.
7.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
About the authors
Pam Sammons is a Professor of Education at the Department of Education,
University of Oxford and a Senior Research Fellow at Jesus College, Oxford.
Previously she was a professor at the School of Education, University of
Nottingham (2004-2009) and a professor at the Institute of Education University of
London (1993- 2004) where she directed the International School Effectiveness &
Improvement Centre (ISEIC) 1999-2004. Her research over more than 30 years has
focused on school effectiveness and improvement, school leadership, teaching
effectiveness and professional development, and promoting equity and inclusion
in education.
Jenni Ingram is an Associate Professor of Mathematics Education at the
Department of Education, University of Oxford and Vice Principal of Linacre
College, Oxford. Previously she was an Assistant Professor at the University
of Warwick and a secondary mathematics teacher in inner-city schools in the
Midlands. Her research focuses on mathematics education and mathematics
teacher education, particularly the role of language in the teaching and learning of
mathematics and the using videos in teacher professional development.
Ariel Lindorff is a Research Fellow in the Oxford University Department of
Education. She completed her DPhil at the University of Oxford, and was previously
a secondary mathematics teacher in inner-city settings in the USA. Her research
focuses on educational effectiveness, improvement and equity, often involving
advanced quantitative methods and/or mixed methods.
Funding for this literature review was provided by the Department for Education
(DfE) as part of the TALIS Video Study. The Organisation for Economic Co-
operation and Development (OECD) led TALIS Video study, is a pioneering,
international study, seeking to improve understanding of which aspects of teaching
are related to pupil learning, and the nature of those relationships. The study
focuses particularly on the teaching of mathematics in secondary schools. Over
approximately one year, more than 750 mathematics teachers in eight countries
(including England) will take part in the study. The study will provide a valuable
opportunity to explore how teachers teach in different countries and contexts.
Education Development Trust, in partnership with Oxford University, manages
England’s involvement in the study on behalf of DfE. Fieldwork is ongoing,
commencing in October 2017 and due to end in July 2018.
5
9.
OBSERVING EFFECTIVE MATHEMATICS TEACHING: A REVIEW OF THE LITERATURE
The quest to understand how best we as educators can teach children and young
people is ongoing. We understand a great deal thanks to the work of many scholars
and academics who have spent decades studying effective teaching and, in no
small way, also to teachers themselves who have been honing their craft over their
careers. Despite this, there is still so much we do not know. A recent Organisation
for Economic Co-operation and Development (OECD) paper described the
‘Supporting teachers has become a top priority across the globe for
the improvement of the quality of our education systems. This renewed
commitment to the teaching profession is based on evidence that teachers are
what makes the greatest difference to learning outside students’ backgrounds,
and that the quality of our school systems is only as good as the quality of
our teachers. A better understanding of what teaching looks like and which
approaches are most effective is not a trivial matter. It is critical, for teaching is
at the heart of a teacher’s role and of the education process.’ 1
Observation is a tool used in teaching and in educational research, albeit in
very different ways in these different disciplines. This review focuses on its use
in research and the study of effective teaching. Observation has tremendous
power to further our understanding about teaching and learning. The education
community is becoming increasingly engaged in the use of video. Simple,
commonplace technologies are helping. This review came about as a result of our
own engagement in an international study, the pilot OECD Teaching and Learning
International Survey (TALIS) Video Study, which uses video to capture and analyse
approaches to teaching mathematics. The OECD report goes on to state:
‘[…] we lack strong evidence about how teaching influences student outcomes
and little is based on actual observation of classroom processes […]. Video-
based research methods now offer an opportunity to understand what teaching
looks like across the globe and, in turn, to enable teachers to learn from their
peers […].’ 2
The tools and approaches that educational researchers employ to understand
teaching through observation are a bit of a mystery to many people, particularly
non-researchers. This review is a clear and concise descriptive summary of the
most tried and tested tools, frameworks and approaches that researchers use to
analyse ‘observed’ teaching. It is an interesting read for anyone involved in the
conduct of observation studies linked to effective teaching, particularly where
mathematics is an area of focus.
Anna Riggall (PhD)
Head of Research,
Education Development Trust
OECD (2018: 2) 2 Ibid.
7
11.
CHAPTER 1: INTRODUCTION
This review examines a range of lesson
observation frameworks designed for and
used in the observation of teaching in
mathematics. This includes frameworks
specifically designed for international
comparisons of teaching practices and
teacher effectiveness, as well as those used
for teaching development.
Classroom observations can be used in a variety of ways, but they are primarily for
the evaluation of teaching, for making comparisons, for professional development
There is a complex
or for a combination of these. There is a complex relationship between teaching
relationship
and learning, and observations of lessons is just one way of examining this
between teaching
relationship. As learning cannot be observed directly, observations usually focus
and learning, and
on identifying particular features of the teaching behaviour and the student
observations of
response. Links are then made with other sources of information, such as student
lessons is just one
attainment and progress measures, student ratings of teaching or lesson artefacts,
way of examining
such as examples of students’ work completed in the lesson.
this relationship
Different frameworks for classroom observations
There is a wide range of observation frameworks available and each is designed
to serve a different purpose. The frameworks included in this review are offered as
examples of this diversity and to exemplify the issues around the design, usability,
validity and reliability of classroom observations. The focus is on frameworks that
have focused specifically on the teaching of mathematics but the review also
includes two generic frameworks used specifically for international comparisons of
the quality and effectiveness of teaching. These two generic frameworks also serve
to illustrate the influence of subject matter on the classroom observation process.
The review excludes frameworks to evaluate the success of particular interventions
or policy initiatives, as these are typically designed to focus on a narrower range
of classroom behaviours and are driven by underlying theories of what ‘good’
teaching might be.
Two key considerations in the design of lesson observation frameworks are
the purpose of the observation and who will be conducting it. For systematic
observations whose purpose is to identify differences between groups of teachers,
for example international comparisons, both validity and reliability are key factors.
9
12.
CHAPTER 1: INTRODUCTION
Similarly, if the purpose of an observation is to make a judgement about the quality
of teaching then reliability is also a key factor. However, as Coe et al.3 point out,
classroom observations that identify teachers as ‘above average’ or ‘below average’
are accurate only about 60% of the time. In value-added measures of teacher
effectiveness (based on changes in student attainment) the issue of measurement
error is acknowledged as important. The use of ‘confidence intervals’ 4 seeks to
address the concept of statistical uncertainty. Measurement error also applies to
observation instruments and this is why inter-rater reliability measures are required
and observations over more than one lesson are desirable (see Hill, Charalambous
and Kraft 5 for an in-depth discussion of the issue of reliability when observing for
teacher quality).
Breadth of focus
Another key consideration in the design of lesson observation frameworks is the
breadth of focus. Too broad a focus and the framework becomes impractical. The
complexity of the classroom means it is unrealistic to try and observe everything
in a lesson. Decisions therefore need to be made about what to observe, and
these decisions are often framed by the purpose of the schedule or the underlying
theoretical basis of what ‘good’ or ‘effective’ teaching might be. There is also the
The Teaching
potential problem of reducing teaching to a checklist of observable practices,
and Learning
as these cannot take into account the decision-making process behind these
International
behaviours, which are particularly relevant when considering the role of teachers’
Survey (TALIS)
pedagogic subject knowledge. 6 These decisions may be an essential part of what
Video Study 2018
makes the practices effective and be of particular relevance to mathematics
is not designed
teaching, since teachers’ subject knowledge has been found to be more strongly
to judge teachers
linked to variations in student outcomes than it is in some other subjects.7
but rather to be a
source of evidence
based research on
typical patterns
Reliability refers to the extent to which a judgement about a lesson could be of teaching and
replicated. A wide range of factors can affect the reliability of a classroom their associations
observation framework score, including the topic being taught, the individual with a range of
teacher and the observer. However, the extent to which the reliability of an outcomes
observation framework matters depends on the use of the framework. The issue
of reliability is particularly important if observations are used in high-stakes
judgements about individual teachers, such as in relation to teacher promotion or
pay increases, rather than as a research instrument to establish variation in practice
and associations with student outcomes. The Teaching and Learning International
Survey (TALIS) Video Study 2018 is not designed to judge teachers but rather to
be a source of evidence based research on typical patterns of teaching and their
associations with a range of outcomes, including student attainment.
To improve the reliability of lesson observation frameworks, the focus tends to
be on observable behaviours that are ‘low inference’ – that is, the observer is
recording whether something occurs or not without making any judgement about
Coe et al. (2014) 4 95% confidence intervals mean that, if we were to measure teacher effectiveness 100 times, 95 of the intervals generated would contain the true, unobservable, measure
of teacher effectiveness 5 Hill, Charalambous and Kraft (2012) 6 Hewitt (2005) 7 see Hill et al. (2008)
13.
CHAPTER 1: INTRODUCTION
whether the behaviour is ‘good’ or not. Examples including counting the number
of questions asked or the amount of time spent on different activities. ‘High-
inference’ items require the observer to make judgements about what they are
observing, such as on the clarity of an explanation or the sequencing of the subject
One key way
One key way of improving the reliability of lesson observation scores is to observe of improving
more than one lesson with a specific teacher, covering a range of topics and the reliability
classes. This can help in identifying which variations in practice can be attributed of lesson
to the topic taught or the class being taught. Another issue with observing single observation
lessons and using these to evaluate teachers is that there is a risk that lessons are scores is to
performances and not representative of an individual’s practice in general or over observe more
time. Hill et al. 8 recommend that at least three lessons be observed by at least two than one lesson
different observers to enhance the reliability of observations made. with a specific
teacher, covering
Another common way of improving reliability is for multiple observers to observe
a range of topics
each lesson. Videoing lessons enables the observation of lessons from multiple
and classes
perspectives and with multiple observers. However, videos do not necessarily
capture the full range of what was going on in the lesson, with a single camera
frequently focusing only on the teacher. This can mean a loss of information
around the interactions between students during the lesson, as well as some
interactions between the teacher and the students. Nevertheless, the use of high-
quality frameworks and trained observers and pooling the findings of observations
of multiple lessons by a range of observers can all help improve the reliability of
the observation process.9
These issues of reliability are less of a concern when observation is being used as
a professional development tool. In these situations, it is the feedback that follows
the lesson observation that matters more.10 Teachers often judge observation
feedback to be most useful when it is a subject specialist who can offer advice on
how to improve the lesson who conducts it.11 Evans, Jones and Dawson12 found
that the usefulness of feedback was dependent on whether the observer was a
mathematics specialist or not, and that these judgements were based on the advice
observers offered on how to improve a lesson. Mathematics specialists offered
substantially more suggestions for improvement, with around half of these relating
specifically to the subject-centred aspects of the lesson. In addition, teachers have
identified peer observation as less threatening and as offering a basis for mutual
learning and support to improve practice.13
Validity is a particularly challenging issue in the development of lesson observation
frameworks, especially given the wide range of purposes for which observations
are used. The validity of a framework relates to the extent to which it is measuring
what it is intended to measure. Many mathematics-specific schedules focus on the
observation of teacher subject knowledge and these frameworks can be correlated
with written assessments of this same subject knowledge, although in practice
these correlations are not strong. For example, the Mathematical Quality of
Hill, Charalambous and Kraft (2012) 9 Strong, Gargani and Hacifazlioglu (2011) 10
Coe et al. (2014) 11
Wragg et al. (2002) 12
Evans, Jones and Dawson (2014) 13
Muijs and Reynolds (2005)
11
14.
CHAPTER 1: INTRODUCTION
Instruction (MQI) framework is largely based on this relationship between teacher
performances on written assessments of their subject knowledge and observations
of these same teachers using these schedules. However, these processes for
examining validity are dependent on a range of factors that make them difficult
to carry out. To compare the scores you need an existing instrument that was
designed to measure the same constructs and that has also been extensively tested
for its own validity and reliability. These sorts of studies all need larger sample
sizes than are typically associated with observation studies if they are to generate
confidence in the statistical analyses.
Lesson observation frameworks can also be validated through comparison
with other existing frameworks. If the two frameworks are measuring the same
construct, such as the quality of mathematics teaching, then their scores will
correlate. However, it can be difficult to find frameworks that measure the same
constructs. For example, one study found a low correlation between frameworks
specifically focusing on mathematics teaching and more general frameworks.14
This therefore suggests these frameworks are measuring distinct constructs.
Observation frameworks
This review presents frameworks designed specifically for the comparison
of teacher and school effectiveness internationally. The first of these, the
International System for Teacher Observation and Feedback (ISTOF),15 is a general
framework examining teacher effectiveness, with over 20 countries, including the
UK, involved in its development. This particular framework has been also been This review
used widely for studies of teacher effectiveness within the UK. presents three
The second framework, Quality of Teaching (QoT),16 was similarly designed to frameworks
examine primary teaching quality across four countries, including England, and is designed
also used in teacher effectiveness studies within the UK, often alongside the ISTOF. specifically for
the comparison
The final framework was specifically designed for the observation of mathematics of teacher
lessons, as part of the Mathematics Education Traditions of Europe (METE) project,17 and school
which involved five European countries, including England. The project also effectiveness
focused on the teaching of three particular topics to students aged ten to 14. The internationally
existing research using these three frameworks includes measures of reliability and
validity, making them suitable for research into the quality of teaching.
This review also considers three other frameworks, each designed specifically for
the observation of mathematics lessons. The first, the Knowledge Quartet (KQ),18
developed as part of the Subject Knowledge in Mathematics (SKIMA) research
programme run by the University of Cambridge, 19 was designed specifically to
support the development of mathematics teaching in primary schools in the UK,
particularly among student teachers. The basis of this framework is teacher subject
knowledge and how it influences the teaching of mathematics. It is now used
more broadly by researchers and teacher educators, but remains a tool for teacher
professional development.
Kane and Staiger (2012) 15
Kyriakides et al. (2010) 16
Van de Grift (2007) 17
Andrews (2007) 18
Rowland, Huckstep and Thwaites (2005) 19
See http://www.knowledgequartet.org/introduction/
for further information
15.
CHAPTER 1: INTRODUCTION
A second framework, referred to as the Watson framework, developed in the UK,20
again was designed with a focus on developing mathematics teaching but this time
at the secondary level. This framework starts from the position of identifying what
aspects of mathematics are being made available to students through the teaching,
rather than focusing on teaching characteristics.
The final framework originates from the US and focuses on the evaluation of
mathematics teaching. The Mathematical Quality of Instruction (MQI) framework,21
has been developed over several years by a team at Harvard, led by Heather
Hill, and is now used in a variety of countries to evaluate mathematics teaching.
Similarly to the Watson Framework, the focus is on mathematical content and how
it is made available to students; similarly to KQ, there is a focus on teacher subject
knowledge. However, in contrast with both KQ and the Watson framework, the MQI
was designed to provide scores for individual mathematics teachers on a number
of discrete dimensions of their mathematics teaching. It is also one of the few
mathematics-specific observation frameworks where there has been considerable
research examining both its validity and its reliability, both within the US and in
other cultural contexts, by means of triangulation of evidence from tests of teacher
subject knowledge, observations of practice and ‘value-added’ measures of
student attainment outcomes.22
Watson (2007) 21
See https://cepr.harvard.edu/mqi for further information 22
See, for example, Hill, Rowan and Ball (2005); Kane and Staiger (2012)
13
16.
Chapter 2
Tools for the
observation of
effective teaching
17.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
The International System for Teacher
Observation and Feedback was developed
by researchers working across 20
countries, including the UK, to specifically
explore the effectiveness of teaching
internationally. 23
The International System for Teacher Observation The ISTOF
and Feedback protocol includes
21 indicators
The ISTOF system schedule is based on recording the extent to which the
grouped
observer agrees that a particular item's description has been observed. The
into seven
fact that this framework offers feedback for teachers is based on research and
components of
expert opinion from more than 20 countries, and that it has been used in a
effective teaching
variety of educational contexts, makes it a useful and reliable framework for
observing teaching.
This schedule has been used in effectiveness studies within England,24
combined with the QoT schedule below. The ISTOF has also been used in
England in the evaluation of Teach First.25
The ISTOF protocol includes 21 indicators grouped into seven components
of effective teaching, as table 1, overleaf, shows. There are 45 items in total,
which are rated on a five-point Likert scale, from strongly agree (5) to strongly
disagree (1), with the option of indicating that it was not possible to observe
some features when they were not relevant to or observable in the particular
classroom setting.
Kyriakides et al. (2010) 24
Day et al. (2008); Sammons et al. (2014) 25
Muijs, Chapman and Armstrong (2012)
15
18.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 1: ISTOF PROTOCOL 26
Category Indicator Item
Assessment and The teacher gives explicit, • The teacher makes explicitly clear why an answer is correct or not
evaluation detailed and constructive
• The teacher provides his/her feedback on the answers given by the students
feedback
Assessment is aligned with • Assignments given by the teacher are clearly related to what students learned
goals and objectives
• The teacher explains how assignments are aligned to the learning goals of the
lesson
Differentiation and The teacher creates an • Students communicate frequently with one another on task-oriented issues
inclusion environment in which all
• Students actively engage in learning
students are involved
The teacher takes full account • The teacher makes a distinction in the scope of the assignments for different
of student differences groups of students
• The teacher gives additional opportunities for practice to students who need them
Clarity of instruction The teacher shows good • The teacher regularly checks for understanding
communication skills
• The teacher communicates in a clear and understandable manner
There is clear explanation • The teacher clearly explains the purposes of the lesson
of purpose
• The teacher asks students to identify the reasons why specific activities take place
in the lesson
Lessons are well structured • The teacher presents the lesson with a logical flow that moves from simple to more
complex concepts
• The teacher implements the lesson smoothly, moving from one stage to another
with well-managed transition points
Instructional skills The teacher is able to engage • The teacher provides sufficient wait time and response strategies to involve all
students types of students
• The teacher gives assignments that stimulate all students to active involvement
The teacher possesses good • The teacher poses questions that encourage thinking and elicit feedback
questioning skills
• The length of the pause following questions varies according to the difficulty level
of questions (e.g. a question calling for application of abstract principles requires a
longer pause than a factual question)
The teacher uses various • The teacher uses a variety of instructional strategies during the lesson
teaching methods and
• The teacher uses different strategies for different groups of students
strategies
Promoting active The teacher helps students • The teacher invites students to use strategies that can help them solve different
learning and developing develop problem-solving and types of problems
metacognitive skills meta-cognitive strategies
• The teacher invites students to explain the different steps of the problem-solving
strategy they are using
• The teacher explicitly provides instruction in problem-solving strategies
The teacher gives students • The teacher encourages students to ask one another questions and to explain their
opportunities to be active understanding of topics to one another
learners
• The teacher gives students the opportunity to correct their own work
The teacher fosters critical • The teacher motivates the students to think about the advantages and
thinking in students disadvantages of certain approaches
• The teacher asks the students to reflect on the solutions/answers they give to
problems or questions
• The teacher invites the students to give their personal opinion on certain issues
The teacher connects • The teacher systematically uses material and examples from the students’ daily life
material to students’ to illustrate the course content
real-world experiences
• Students are invited to give their own examples
Adapted from Kyriakides et al. (2010)
19.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 1: ISTOF PROTOCOL (CONTINUED) 26
Category Indicator Item
Classroom climate All students are valued • The teacher demonstrates genuine warmth and empathy towards all students in
the classroom
• The teacher shows respect for the students in both his/her behaviour and the use
of language
The teacher initiates active • The teacher creates purposeful activities that engage every student in productive
interaction and participation work
• The teacher’s instruction is interactive (lots of questions and answers)
The teacher interacts with all • The teacher gives turns to and/or involves those students who do not voluntarily
students participate in classroom activities
• The teacher seeks to engage all students in classroom activities
The teacher communicates • The teacher praises students for effort towards realising their potential
high expectations
• The teacher makes clear that all students know that he/she expects their best
efforts in the classroom
Classroom management Learning time is maximised • The teacher starts the lesson on time
• The teacher makes sure students are involved in learning activities until the end of
the lesson
• Actions are taken to minimise disruption
Clear rules are evident • There is clarity about when and how students can get help
• There is clarity about what options are available when the students finish their
assignments
Misbehaviour and disruptions • The teacher corrects misbehaviour with measures that fit the seriousness of the
are effectively dealt with misconduct (e.g. s/he does not overact)
• The teacher deals with misbehaviour and disruptions by referring to the
established rules of the classroom
Adapted from Kyriakides et al. (2010)
17
20.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
The Quality of Teaching framework The QoT
framework was
The QoT framework was developed by school inspection teams from four
developed by
countries, including England,27 to inspect the quality of teaching across these
school inspection
countries in primary schools. 28 This is a value-based framework with high-
teams from
inference codes requiring the observer to balance the strengths and
four countries,
weaknesses of different features of the classroom practice being observed.
including
The observer awards an overall grade designed to reflect an overall judgement
England, to
of lesson quality. The initial development of this framework included studies
inspect the quality
that examined the reliability, inter-rater reliability and validity of the observation
of teaching across
framework, specifically focusing on the teaching of mathematics in primary
these countries in
schools. Subsequently, the use of the framework has been extended to other
primary schools
curriculum areas and to secondary lessons within the UK.29
The QoT framework requires trained observers to make professional judgements
about the practice being observed. It draws on the professional judgement
systems used by inspectorates in multiple countries, alongside the educational
effectiveness research literature. It has been tested in a number of European
countries, which has shown that the measures are reliable and mostly scalar
equivalent between different countries.30
The framework itself has six quality characteristics, and each item within these
includes examples of ‘good practice’ to improve the reliability of the judgements
observers make, as table 2 shows. When the completing the table during a lesson
observation, observers must:31
• Score each item on a 1–4 scale depending on the balance of strengths and
weaknesses. The observer places a teacher on the scale according to the
following:
1 = predominantly weak;
2 = more weaknesses than strengths;
3 = more strengths than weaknesses;
4 = predominantly strong.
The observer must score 3 only when all good practice examples (if applicable)
are really observed.
• Circle the correct answer:
0 = no, I didn’t observe this;
1 = yes, I have observed this.
27
Also Belgium, France and the Netherlands 28
Van de Grift (2007) 29
See Day et al. (2008); Sammons et al. (2014) 30
Van de Grift (2013) 31
Van de Grift (2007:148–152)
18
21.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING 32
Rate Indicators: The teacher… Observed
Efficient classroom • gives a well-structured lesson 1234 • ensures clearly recognisable components in the lessons 01
management (lesson structure)
• ensures the orderly progression 1234 • ensures entering and leaving the classroom take place in an 01
of the lesson orderly manner
• intervenes in a timely and appropriate way to any order
disruptions
• acts as a ‘watchdog’ for agreed codes of behaviour and rules
• uses learning time efficiently 1234 • ensures there is no loss of time at the start, during or at the 01
end of the lesson
• ensures there are no ‘dead’ moments
• ensures the students are not left waiting
• ensures efficient classroom 1234 • makes clear which lesson materials should be used 01
management
• ensures the lesson materials are ready to use
• ensures the lesson materials are adapted to the level and
experience of the students
Safe and stimulating • ensures a relaxed atmosphere 1234 • addresses the children in a positive manner 01
learning climate
• reacts with humour, and stimulates humour
• allows children to make mistakes
• promotes mutual respect 1234 • encourages students to listen to one another 01
• intervenes when students are being laughed at
• takes (cultural) differences and idiosyncrasies into account
• supports the self-confidence of 1234 • feeds back on questions and answers from students in a 01
students positive way
• expresses positive expectations to students about what they
are able to take on
• shows respect for the students in 1234 • allows students to finish speaking 01
behaviour and language use
• listens to what students have to say
• makes no role-confirming remarks
• ensures cohesion 1234 • honours the contributions made by students 01
• ensures solidarity between students
• ensures events are experienced as group events
• stimulates the independence of 1234 • allows students to work independently on another 01
students assignment or to take up an individually selected task after
completing an assignment
• allows students to work with self-correcting materials
• has students working on daily and weekly tasks
• promotes cooperation between 1234 • provides opportunities for students to help one another 01
students
• gives assignments that incite cooperation
• gives students the opportunity to play together or to carry
out assignments together
Clear instruction • clarifies the lesson objectives at 1234 • informs students at the start of the lesson about the aims of 01
the start of the lesson the lesson
• clarifies the aim of the assignment and what the students
will learn from it
• evaluates whether the objectives 1234 • verifies and/or evaluates whether the aims of the lesson 01
have been achieved at the end of have been achieved
the lesson
• checks the students’ achievements
Adapted from Van de Grift (2007: 148–152)
19
22.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32
Rate Indicators: The teacher… Observed
Clear instruction • gives clear instructions and 1234 • activates the student’s prior knowledge 01
(continued) explanations
• explains in sequential stages
• asks questions that are understood by the students
• summarises the lesson materials from time to time
• gives clear explanations of 1234 • ensures that every student knows what he/she has to do 01
the learning materials and the
• clearly indicates the materials that can be used as
assignments
learning aids
• gives feedback to students 1234 • checks whether students have understood the lesson 01
materials when he/she is instructing the class
• checks whether students are completing the assignments
correctly
• gives feedback on the way students arrive at their answers
• gives feedback on the social functioning involved in the
completion of the tasks (group work)
• involves all students in the lesson 1234 • gives assignments that stimulate students into active 01
involvement
• poses questions that initiate reflection
• ensures students listen carefully and keep on working
• waits sufficiently long to allow students to reflect after
posing a question
• gives the opportunity to respond to students who don’t put
their hands up
• makes use of teaching methods 1234 • makes use of conversational forms and discussion forms 01
that activate the students
• provides graduated exercises
• permits working in groups/corners
• makes use of information and communication technology
Adaption of • adapts the instruction to the 1234 • allows students who need less instruction to commence 01
teaching relevant differences between with the work
students
• gives extra instruction to small groups or individual students
• does not direct himself exclusively to the middle bracket
• adapts the assignments and 1234 • makes a distinction in the scope of the assignments between 01
processing to the relevant individual students
differences between students
• does not give all students the same time to complete the
assignment
• allows some students to make use of auxiliary materials
Teaching learning • ensures the teaching materials are 1234 • teaches students solution strategies or search and reference 01
strategies oriented towards transfer strategies
• teaches students the use of organisation resources
• promotes the conscious use of what has been learned in
other (different) areas of learning
• stimulates the use of control 1234 • gives attention to estimatory calculation/anticipatory 01
activities reading
• has solutions related to the context
• stimulates the use of alternative solutions
• provides interactive instruction 1234 • facilitates mutual interaction between students… ensures 01
and activities interaction between pupils and the teacher
Adapted from Van de Grift (2007: 148–152)
23.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 2: QOT FRAMEWORK – LESSON OBSERVATION FORM FOR EVALUATING THE QUALITY OF TEACHING (CONTINUED) 32
Rate Indicators: The teacher… Observed
Involvement • ensures there is good individual 1234 • ensures pupils actively listen to the instructions 01
of pupils involvement by the pupils
• ensures pupils take part in learning/group discussions
• ensures pupils work on the assignment in a concentrated,
task-focused way
Final judgement The overall quality of teaching is 1234
judged as:
The Mathematics Education Traditions of Europe project
The METE observation framework developed out of a study comparing
mathematics teaching in five European countries: England, Finland, Flanders
Belgium, Hungary and Spain. The schedule was developed through live
observations and then used video recordings of lessons for the main analyses. The
lessons focused on the teaching of specific topics with students aged ten to 14:
percentages, polygons and linear equations. The main focus of the study was on
how mathematics teachers structured students’ opportunities for learning.33 The
framework consists of three broad categories, each containing several foci, which
were designed to be easily applied across all observers in all countries involved.
Each category was designed to be low inference and to address observable
behaviours in the lessons. The first category refers to the mathematical foci or
observable learning outcomes, as table 3 shows.
TABLE 3: METE MATHEMATICAL FOCI 34
Mathematical foci Description – the teacher is seen to emphasise or encourage:
Conceptual the conceptual development of his or her students
Derivational the process of developing new mathematical entities from existing knowledge
Structural the links or connections between different mathematical entities, concepts, properties, etc.
Procedural the acquisition of skills, procedures, techniques or algorithms
Efficiency pupils’ understanding or acquisition of processes or techniques that develop flexibility, elegance or critical
comparison of working
Problem solving pupils’ engagement with the solution of non-trivial or non-routine tasks
Reasoning pupils’ development and articulation of justification and argumentation
The second category for observation focuses on the contexts in which the teachers
posed the tasks. It has two dimensions: (1) whether the context was related to the real
world or not and (2) whether the data or information used was genuine or invented
by the teacher. In this way, the assessment of mathematics classroom activity can be
carried out using a two-dimensional grid, as shown in table 4, overleaf.
Adapted from Van de Grift (2007: 148–152) 33
Andrews (2009) 34
Adapted from Andrews (2007: 501)
21
25.
CHAPTER 2: TOOLS FOR THE OBSERVATION OF EFFECTIVE TEACHING
TABLE 4: METE CONTEXT 35
Dimensions Example
The task is explicitly related to • The task of calculating the cost of decorating a hypothetical room is related to the real world but is located in a fantasy
the real world and based on of data – the dimensions of the room, the costs of paper, for example.
data or entities invented by the
• Revising the cost of a pair of hypothetical trousers after a sale reduction.
The task is explicitly not related • An invitation to solve the equation x2–3x+1=0 is not based in the real world and the data or entity – the equation itself
to the real world and based on – is not the product of a student’s own activity.
data or entities invented by the
• Many text-based questions or exercises would fall into this category.
The task is explicitly related to • Testing statistical hypotheses derived from real data collected by students.
the real world and based on
• Calculating the cost of manufacturing a desk by measuring the desk. It is the act of measurement, which creates
genuine data or entities
genuine data, that feeds back into the real world, as it addresses the cost of making the desks.
The task is explicitly not related • Exploring the minimum value of a quadratic expression of the student’s choice has no explicit relation to the real world,
to the real world and based on but the data – the choice of the individual student – is real.
genuine data or entities
• An invitation to students to measure the length of their desks for no other purpose than to practise the skills of
measurement. The task is not explicitly related to the real world because it does not feed back into it, but it is located in
real-world, genuine data. In this scenario, the real world provides a background context for the task.
The final category concerns teacher strategies, or ‘mathematical didactics’, that
might be used to facilitate students' learning of mathematics, as table 5 shows.
These categories were used both when the teacher was working with a class as a
whole and when the students were working individually or as a small group.
TABLE 5: METE TEACHING STRATEGY 36
Teaching strategy Description
Activating prior knowledge Focuses students' attention on mathematical content covered earlier in their careers via a period of revision as
preparation for activities to follow.
Exercising prior knowledge Focuses students’ attention on mathematical content covered earlier in their careers via a period of revision unrelated to
any activities that follow.
Explaining Explains an idea or solution. This could include demonstration, explicitly telling or the pedagogic modelling of higher-
level thinking. In such instances, the teacher is the informer with little or no student input.
Sharing Engages students in the sharing of ideas, solutions or answers. This could include class discussions, where the teacher's
role is one of manager rather than informer.
Exploring Engages students in an activity, not teacher directed, from which a new mathematical idea is intended to emerge.
This activity could be an investigation or a sequence of structured problems, but in all cases students are expected to
articulate their findings.
Coaching The teacher explicitly offers hints, prompts or feedback to facilitate students’ understanding of or ability to perform tasks
or to correct misunderstandings.
Assessing or evaluating Assesses or evaluates students’ responses to determine the overall attainment of the class.
Motivating Addresses students’ attitudes, beliefs or emotional responses towards mathematics.
Questioning The teacher explicitly uses a sequence of questions, perhaps Socratic, so as to lead students to construct new
mathematical ideas or clarify or refine existing ones.
Differentiation Attempts to treat students differently in terms of the kind of activities performed, materials provided and/or the expected
outcome to make instruction optimally adapted to the students’ characteristics and needs.
Adapted from Andrews (2007) 36
Adapted from Andrews (2007: 503)
23
26.
Chapter 3
The use of
frameworks for
research in the UK
27.
CHAPTER 3: THE USE OF FRAMEWORKS FOR RESEARCH IN THE UK
The ISTOF, QoT and METE frameworks,
have all been used by researchers in the
UK to examine teaching, and specifically
mathematics teaching, through
observations of teaching practice.
These studies often use the frameworks in conjunction with other sources of data,
such as teacher interviews and student questionnaires, in order to gain a fuller
The effective
description of more effective teaching. The findings from these studies relate
classroom practice
closely to the existing literature on effective teaching and learning.
study used the
The effective classroom practice study used the ISTOF and QoT,37 alongside teacher ISTOF and QoT,
questionnaires and interviews, school leader interviews and pupil questionnaires alongside teacher
and interviews, to establish a multidimensional picture of effective classroom questionnaires
practice. The frameworks specifically identified several core characteristics of and interviews,
more effective teaching. Specifically, the ISTOF identified clear and coherent school leader
lessons with a supportive learning climate; engaging students with assignments interviews
and activities; positive classroom management; purposive learning; and quality and pupil
questioning and feedback for students. The more effective teachers also questionnaires
scored very highly on the QoT characteristics of a supportive lesson climate; and interviews
proactive lesson management; well-organised lessons with clear objectives; and
environmental and teacher support.
The inspiring teachers study took a mixed-methods approach to characterising
inspiring teachers,38 again using both the ISTOF and QoT, combined with
qualitative observations, teacher and school leader interviews, and pupil
questionnaires. The sample included 17 teachers representing primary and
secondary schools. The comparison between the qualitative observations and the
use of the two quantitative frameworks, the ISTOF and QoT, revealed that inspiring
teachers also showed strongly the characteristics of more effective teaching. In
particular, the teachers identified as inspiring scored particularly highly on the
ISTOF in relation to creating a positive classroom climate, classroom management
and clarity of instruction. Similarly, these teachers also scored highly on the QoT
components related to a safe and orderly school climate, effective classroom
layout, clear instruction and effective classroom organisation. As such, while
defining inspiring teachers relies more on ideas, such as student engagement
and enjoyment, than on effectiveness studies that focus on student academic
outcomes, the findings of this study conclude that inspiring teachers are first
and foremost highly effective teachers.
The ISTOF has also been used by Muijs, Chapman and Armstrong39 to explore the
effectiveness of Teach First teachers (an alternative teacher certification programme
in England). Similarly to the studies above, this framework was used in conjunction
See Day et al. (2008): Kington et al. (2012) 38
Sammons et al. (2014) 39
Muijs, Chapman and Armstrong (2012)
25
28.
CHAPTER 3: THE USE OF FRAMEWORKS FOR RESEARCH IN THE UK
with interviews with teachers and school leaders, and teacher questionnaires. The
Teach First teachers demonstrated high levels of the behaviours in the framework
that are considered indicators of more effective teachers and they also scored
similarly to those teachers observed during the design of the ISTOF framework. 40
Conclusion: Comparing the ISTOF, QoT and METE frameworks
Whilst the ISTOF and QoT share some similarities in terms of their components
and measures, they are both conceptually and practically very different measures
of teaching behaviours. Both schedules show sufficient reliable test results for
use in the two studies above, but the ISTOF framework scores higher on inter-
rater reliability and reliability than the QoT framework. The correlation between
teachers’ overall scores on the ISTOF and the QoT were strong, positive and
statistically significant. Both frameworks provide an overall measure of effective
practice but also distinguish different features of practice that can be used when The METE
giving feedback to the teachers involved. Similarly, combining these frameworks framework
with field notes can contribute to the usefulness of feedback, as they can provide has been used
more detail on student characteristics and prior learning. to explore
The studies that have used these frameworks in combination suggest that there is similarities
an overall concept of teacher effectiveness, but also that there are differentiations and differences
within this. Consequently, more effective teachers would show both strengths and between
weakness in particular aspects of their practice that might vary over time, with teachers, both
different topics and with different students. The frameworks themselves identify in England and
broad descriptions of more effective practice, but both the effective classroom internationally
practice study and the Inspiring Teachers Study found considerable variation in
the ways that these broad categories of more effective practice were enacted by
the teachers.
The METE framework has been used to explore similarities and differences
between teachers, both in England and internationally. It has not been used to
compare the effectiveness of particular teaching behaviours but rather what
similarities and differences across groups of teachers or across a particular topic,
such as linear equations, can tell us about the learning of mathematics. Similarly to
the Inspiring Teachers Study, whilst some similarities across the broad categories
was observed, there were noticeable differences in the ways that these categories
were observed in teachers’ classroom practice. 41
Kyriakides et al. (2010) 41
Andrews, 2009.
31.
CHAPTER 4: ALTERNATIVE FRAMEWORKS
The three frameworks that follow have
largely been used in different ways from the
three frameworks already discussed.
For example, the Mathematical Quality of Instruction (MQI) framework has been
used widely to evaluate the subject knowledge of mathematics teachers, and the
The validity
number of countries for which it has been validated continues to grow, though
and reliability
there is no study at present showing its validity within the UK. The Knowledge
of the MQI has
Quartet (KQ) has also been used extensively, but as a professional development
been established
tool rather than as a measure of effectiveness or subject knowledge. This contrast
in several
mimics the purposes for which these two frameworks were designed: both were
studies through
initially designed for the observation of primary mathematics lessons but their use
comparison
has extended to secondary mathematics. The Watson framework was specifically
with written
designed for the observation of secondary mathematics classrooms and again has
assessments of
been used predominantly as a professional development tool. The validity and
mathematics
reliability of the MQI has been established in several studies42 through comparison
teachers’ subject
with written assessments of mathematics teachers’ subject knowledge. However,
knowledge
these validity and reliability are not appropriate measures of the usefulness to
professional development. This instead relies on how useful teachers and teacher
educators have found the frameworks in developing their practice.
The Knowledge Quartet
KQ was developed as a framework to support ‘productive discussion of
mathematics content knowledge between teacher educators, trainees and
teacher-mentors’. 43 It was designed as a framework both for lesson observation
and for mathematics teaching development. The focus is on mathematics
subject knowledge and it is designed to develop both mathematics teaching and
mathematics teacher knowledge.
This framework was initially designed through working with primary student
teachers and their university tutors and mentors, and through the analysis of
videos of teaching on teaching practice, but it is now widely used for professional
development purposes at all levels of education.
As table 6 outlines, there are four aspects to KQ. The first category – foundation
knowledge – underpins the other categories, as it focuses on the knowledge and
beliefs of the teacher, with the other categories focusing on the application of that
knowledge in teaching. These categories are not mutually exclusive and episodes
within a lesson can be understood in terms of more than one of them.
For example, ‘a contingent response to a pupil’s suggestion might helpfully
connect with ideas considered earlier’. 44
See, for example, Hill et al. (2008) 43
Rowland, Huckstep and Thwaites (2005: 256) 44
Ibid. (259)
29
32.
CHAPTER 4: ALTERNATIVE FRAMEWORKS
TABLE 6: THE KNOWLEDGE QUARTET 45
Category Description
Foundation Propositional knowledge and beliefs concerning:
• the meanings and descriptions of relevant mathematical concepts, and the relationships that exist between them;
• the different factors that research has shown to be significant in the teaching and learning of mathematics;
• the ontological status of mathematics and the purposes of teaching it.
Contributory codes: awareness of purpose; identifying errors; overt subject knowledge; theoretical underpinning of
pedagogy; use of terminology; use of textbook; reliance on procedures.
Transformation Knowledge-in-action revealed in deliberation and the choices made in planning and teaching.
The teacher transforms and presents his or her own meanings and descriptions in ways designed to enable students to
learn. These could include the use of powerful analogies, illustrations, explanations and demonstrations.
The choice of examples made by the teacher is especially visible:
• for optimal acquisition of mathematical concepts, procedures or essential vocabulary;
• for confronting common misconceptions;
• for the justification (by generic example) or refutation (by counter-example) of mathematical ideas.
Contributory codes: choice of representation; teacher demonstration; choice of examples.
Connection Knowledge-in-action revealed in deliberation and choice in planning and teaching.
Within a single lesson, or across several lessons, the teacher unifies the subject matter and draws out coherence with
respect to:
• connections between different meanings and descriptions of particular concepts or between alternative ways of
representing concepts and conducting procedures;
• the relative complexity and cognitive demands of mathematical concepts and procedures, by attention to sequencing of
the content.
Contributory codes: making connections between procedures; making connections between concepts; anticipation of
complexity; decisions about sequencing; recognition of conceptual appropriateness
Contingency Knowledge-in-interaction revealed through the ability of the teacher to ‘think on his/her feet’ and respond
appropriately to the contributions made by students during a teaching episode.
This could be seen in the teacher’s willingness to deviate from his/her own agenda, when to develop a student’s
unanticipated contribution:
• might be of special benefit to that pupil; or
• might suggest a particularly fruitful avenue of enquiry for others.
Contributory codes: responding to children’s ideas; use of opportunities; deviation from agenda
Mathematical Quality of Instruction
The MQI was developed by Heather Hill and colleagues at the University of
Michigan and Harvard University to reliably measure several dimensions of the
work teachers do with students around mathematical content. The MQI is based on
a theory of instruction, existing literature on effective instruction in mathematics
and an analysis of nearly 250 videotapes of US teachers and teaching. This means
that the design is flexible enough to consider the variety of mathematics teaching
that occurs in classrooms. The MQI is based on the premise that the mathematical
work that occurs in classrooms is distinct from generic features of teaching, such
as classroom climate.
Adapted from Rowland, Huckstep and Thwaites (2005: 265–266)
33.
CHAPTER 4: ALTERNATIVE FRAMEWORKS
The MQI uses the three key relationships widely used in mathematics education
research, often referred to as ‘the didactic triangle’. 46 These are the relationships:
between the teacher and the mathematics; between the teacher and the students;
and between the students and the mathematics, as illustrated in table 7. The
framework provides separate teacher scores for five different dimensions,
which can each be used to assess these relationships. These dimensions are
the richness of the mathematics; errors and imprecision; working with students
and mathematics; student participation in meaning making and reasoning; and
connections between classroom work and mathematics.
The framework uses video recordings of lessons. Each recorded lesson is then
divided into roughly equal length (e.g. 5 or 7.5 minute) segments for scoring by two
independent raters. A score is given for each of these five MQI dimensions and the
raters also each give the whole lesson an overall MQI score.
TABLE 7: MQI CONSTRUCTS 47
Construct Description
Teacher–content relationship Richness of the mathematics
Richness includes two pieces:
(1) attention to the meaning of mathematical facts and procedures and
(2) engagement with mathematical practices and language.
Meaning making includes explanations of mathematical ideas and drawing connections among different mathematical
ideas or different representations of the same idea. Mathematical practices are represented by multiple solution methods,
where more credit is given for comparisons of solution methods for ease or efficiency; by developing mathematical
generalisations from examples; and by the fluent and precise use of mathematical language.
Errors and imprecision
This captures whether the teacher makes major errors indicating gaps in mathematical knowledge; whether the teacher
distorts content through unclear articulation of concepts; and/or whether there is a lack of clarity in the presentation of
content or the launching of tasks.
Teacher–student relationship Working with students and mathematics
This investigates whether the teacher accurately interprets and responds to students’ mathematical ideas. It also looks at
whether the teacher can correct student errors thoroughly, with attention to the specific misunderstandings that led to
the errors.
Student–content relationship Student participation in meaning making and reasoning
This captures the ways in which students engage with mathematical content, specifically whether students ask questions
and reason about mathematics; whether students provide mathematical explanations independently or in response to
the teacher’s questions; and/or the cognitive requirements of specific tasks, such as whether students are asked to find
patterns, draw connections or explain and/or justify their conclusions.
Connections between classroom work and mathematics
This explores whether classroom work has a mathematical point, or whether the bulk of instructional time is spent on
activities that do not specifically develop mathematical ideas, for example cutting and pasting or non-productive uses of
time, including transitions or discipline.
Chevallard (1985); Brousseau (1997) 47
Adapted from National Center for Teacher Effectiveness (2012)
31
34.
CHAPTER 4: ALTERNATIVE FRAMEWORKS
Watson’s framework Frameworks have
been designed for
Watson’s framework48 is included as it is different in design from the other
a specific purpose,
frameworks discussed in that it ‘start[s] from mathematics rather than from
such as the subject
teaching’. 49 This framework was designed for use by mathematics teachers, and
knowledge of
particularly student teachers, to improve the teaching of mathematics. Again,
mathematics
the framework focuses on observable teacher behaviours but also includes
teachers or as
considerations of the ‘kinds of shift a learner might be hoped to make during
a professional
mathematical activity’.50 Table 8 presents the ‘dimensions of mathematical
development
pedagogic orientation’, the relevant tasks or prompts that are observable and the
tool, as well as for
shifts required for each dimension.
primary teaching
or secondary
teaching
These three frameworks all focus on the particular features of mathematics
teaching and enable us to observe the mathematical content and its presentation,
as well as some of the more general features focused on in the earlier frameworks.
Each of these frameworks have been designed for a specific purpose, such as the
subject knowledge of mathematics teachers or as a professional development tool,
as well as for primary teaching or secondary teaching. Whilst their use in other
settings or for other purposes has not been validated by research, many teachers
are finding them a useful framework for analysing their own teaching.
Watson (2007) 49
Ibid. (118) 50
Ibid. (120)
+1-315-636-4485
+91 98151-74050 
