Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


Rubrics and Exemplars: Evaluation and Calibration

No description

junifer abatayo

on 17 October 2012

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Rubrics and Exemplars: Evaluation and Calibration

Rubrics and Exemplars:
Calibration Rubrics Rubrics Types of Rubrics Holistic Holistic Holistic Marking Advantages
of Holistic Marking set of criteria that clearly defines
and presents a range of acceptable
and unacceptable performance
of students. - powerful tools for teaching
and assessment
- Metler (2001) defines rubrics
as rating scales used with
performance assessments
- scoring guides
- evaluation criteria with
pre-established performance
assessment Analytic and Holistic means the scorer or rater
records a single impression
of impact of the performance
as a whole
(McNamara, 2000:43) - marker's or rater's total impression
of the essay or piece of writing
- impressionistic
- global integrative rating
- quick and reliable if 3 or 4 people
mark each script - general rule of thumb for holistic
marking is to mark for two hours
making sure to grade not more than
20 scripts per hour
- most successful using scales of a
limited range (e.g. from 0-6) - it is reliable if done with no time
constraints and teachers receive
adequate training
- quick way of rating in a short
period of time
- students are disadvantage by ONE
lesser ability such as poor
grammar since overall writing
ability is assessed Disadvantages
of Holistic Marking - unreliable if marking is done under short time constraints and with inexperienced, untrained
teachers (Heaton, 1990)
- long essays tend to be unreliable
because of the test's inability to
provide feedback (Cohen, 1994) - washback are always absent;
testers find it difficult to justify
the rationale for the mark
- does not provide of the student's profile of his writing
ability (Hamp Lyons, 1990) Disadvantages
of Holistic Marking Analytic - Raters provide separate assessments
for each of a number aspects of
performance (Hamp-Lyons, 1991)
- Raters mark selected aspect or area
of a piece of writing
- Discrete point marking and focused
holistic marking
- Effective for inexperienced teachers
- More reliable when they have a
larger point range - this type of marking can provide teachers
with a "profile" of their students' strengths
and weaknesses in the area of writing
- reliable if done with a population of
inexperienced teachers who have had little
- gurads against the collapsing of categories
within overall writing ability (Cohen, 1994) Advantages
of Analytic Marking Disadvantages
of Analytic Marking - time consuming
- set of specific criteria
- teachers should be trained
- moderation and calibration are needed
- inter-marker differences are reduced;
increasing validity
- specific areas; content, organization,
grammar, mechanics and
vocabulary Holistic or Analytic Some criteria for Scoring Essay Items
(Borich, 2007) a. Content
b. Organization
c. Process
d. Completeness/Internal
e. Originality/Creativity Validity in its purest form is defined as "the extent to which a test or examination does what it is designed to do" (Alderson, Clapham & Wall, 1996) Assessing Writing: Validity and Reliability Assessing Writing: Validity and Reliability Reliability, is a prerequisite to validity, refers to the overall extent to which a test measures consistently (Bailey, 1998) Consider the following: 1. Does it measure the course objectives?
2. Does it reflect balance and ideal
distribution of or coverage of the
course work?
3. Does it show practicality of the
marking process? Consistent
between markers across a period
of time? Writing Assessment Scales EFL and ESL assessment literature
generally recognizes two types of writing
scales for students' writing proficiency: HOLISTIC MARKING and
(Hamp-Lyons, 1991; Raimes, 1983;
Weir, 1993) Assessing Writing: WHAT and WHY?
RATER AND THE RATING PROCESS, HOLISTIC AND ANALYTICAL SCALES Reliability - moderation, double marking and/or training The Rater and the Rating Process: Bereiter and Scardemalia (1983) explain that essay writing is
probably the most complex constructive act the most human beings are ever expected
to perform. Within the next assessment, McNamara (2000) emphasizes the more general need to take the context
into account in the
following terms. Evaluation of writing,
its processes and result in terms
of score, grade or band may be
detrimental in some way or the other to the
test taker. It would also bring out the idio-
syncrasies or the way raters may differ from
each other - overal leniency, patterns of harshness
in relation to group of students or particular tasks,
the difference of what is consistency and
inconsistency and the way the raters interpret
the rating scale they use (p. 98-100) RECOMMENDATION
"Testers caution, however, that
determining "how" to grade is as important
as "what" to grade. To be fair for students,
they should be given plenty of opportunities
to practice a variety of different writing skills
of varying lengths. In other words, test of
writing should be shorter and more frequent,
not just a "snapshot" approach at midterm
and final exams'.
(Coombe & Evans, 2009)
Full transcript