OHRCAT Presentation

The Oregon Healthy Relationships Curriculum Assessment Tool

The OHRCAT was created in 2015 by The Oregon Attorney General's Sexual Assault Task Force (AGSATF) as a tool to evaluate healthy relationships and violence/abuse prevention curricula based on state and national standards. It is modeled after the CDC's Health Education Curriculum Assessment Tool (HECAT) and Sexual Health Education Curriculum Assessment Tool (SHECAT).

This is the OHRCAT

There are three different tools, an elementary school OHRCAT (grades K-5), a middle school OHRCAT (grades 6-8) and a high school OHRCAT (grades 9-12). Each tool contains the performance indicators that are supposed to be met by the completion of the respective grades.

Each OHRCAT has several parts...

1. Core Concepts

2. Reviewer Guidelines

3. An Overview of the Criteria

4. The Curriculum Review Forms

Core Concepts

This section of the OHRCAT includes important definitions and frameworks that reviewers apply when reviewing curricula. This helps ensure all reviewers are approaching each curriculum from a shared foundation. Some highlights in this section include:

• A dynamic definition of "healthy relationship" informed by Oregon law as well as best-practice, and;
• Key elements of an effective healthy relationships curriculum

Reviewer Guidelines and Criteria

This section provides reviewers with some direction on resources to utilize prior and during review, as well as strategies we ask all reviewers use in completing the OHRCAT. This helps us ensure some uniformity in reviewer approaches.

We will spend time learning about the criteria in detail in 'The Standards' section of this presentation.

The Forms

These are the documents that each reviewer will complete and return to AGSATF at the completion of the review! They include:

• Accuracy Analysis Chart
• Acceptability Analysis Chart
• Summary Score Sheet
• Individual Standard Score Sheets

Accuracy and Acceptability

Although the Accuracy and Acceptability charts look relatively similar, they serve different purposes. The Accuracy chart is to document any errors or inaccuracies in the curriculum. If information is wrong, not up to date, not reflective of current definitions, etc. reviewers document those things here. The Acceptability chart is to document when there are components of a curriculum that don't align with the core criteria in section one, current best-practice, or community norms. These two charts help us when we make our individualized curriculum recommendations.

Summary Score Sheet

The summary score sheet looks the same for all three OHRCATs. This is where you enter your total scores for each standard. This is also a place to document any notes on the curriculum.

Individual Standard Score Sheets

These score sheets make up the bulk of the OHRCAT. With the exception of Standard 1, each standard will have two scores. The first score is based on what students are supposed to learn (color coded with a blue bar at the top of the score sheet). The second score is whether or not the curriculum includes activities for students to practice those skills (color coded with a yellow bar at the top of the score sheet).

Individual Standard Score Sheets

This is where reviewers document how a curriculum meets the state and national standards. Here is Standard 8 of the High School OHRCAT. As reviewers read a curriculum they will go through the OHRCAT individual standard score sheets and mark off when a performance indicator is met. Then they will total up the indicators marked, review what score category it would fall under, then transfer that score to the summary score sheet (as seen on a previous slide).

Individual Standard Score Sheets

After completing the learning assessment of each standard, a reviewer will then use the skills practice assessment to assess whether or not students had the opportunity to practice the skills. This is the skills practice assessment for standard 8. Reviewers will document the instances of skills practice based on the performance indicator for that standard (as listed in the learning assessment) as well as whether the curriculum provides opportunities for self evaluation. This score will then be transferred to the summary score sheet as well.

Special Consideration: K-5 OHRCAT

On the K-5 OHRCAT, performance indicators are divided by grade level.You will see three categories: All Grades, K-2, and 3-5. Elementary curricula are often broken up in these grade bands. When reviewing, only evaluate a lesson based on the performance indicators of the grade level it is written for.

For example: If you are reading a 2nd grade lesson, you would only look at the performance indicators listed under "All Grades" and "K-2".

Now that we've looked at the tool, let's go ahead and look in more detail at the standards and performance indicators!

The Standards

The standards along with the performance indicators listed under each standard, make up 'The Criteria' described at the beginning of each OHRCAT. There are 8 Standards included in each of the OHRCATs. They are based on Oregon's Health Education Standards as well as National Health Education Standards. These Include:

• Standard 1: Core Concepts
• Standard 2: Analyzing Influences
• Standard 3: Accessing Information
• Standard 4: Interpersonal Communication
• Standard 5: Decision Making
• Standard 6: Goal Setting
• Standard 7: Self-Management
• Standard 8: Advocacy

Performance Indicators

Under each of the 8 standards is a list of performance indicators. These indicate the concepts and skills that students are supposed to learn in order for the standard to be met. Some of these are knowledge based, like Standard 1, and some of these are skills based, like Standard 8.

Performance Indicators

Each OHRCAT includes performance indicators from various sources including Oregon's health education standards, and may include

Nation-wide education benchmarks and extra criterion based on the core concepts listed at the beginning of each OHRCAT. Each of the performance indicators has a code that designates where it's from and where to find it.

OHRCAT Criteria

High School (Grades 9-12) Criteria

Elementary School (Grades K-5) Criteria

Middle School (Grades 6-8) Criteria

The OHRCATs use criteria (performance indicators) from three sources:

• The Oregon Health Education Standards and Performance Indicators (OHESAPI)
• Extra Criterion based on the Core Concepts described at the beginning of the OHRCAT (EC)
• Nation-Wide Benchmarks from the CDC's Health Education Curriculum Assessment Tool (HECAT and NSBS)

Numbering of Oregon's Performance Indicators

There are four parts of each Oregon performance indicator number.

• For Oregon, these include: HE (for health education), a number signifying which standard (1 through 8) the performance indicator falls under, a number corresponding to the grade level, and then a number for the standard.

That means that this performance indicator is from Oregon's Health Education Standards. It is from the third standard (Accessing Information). It is a high school level indicator, needing to be completed by 12th grade, and it is indicator number 7.

We can tell from the number that this is from Oregon's Health Education Standards. It is from the fifth standard (Decision Making). It is a middle school level indicator, needing to be completed in 6th and 7th grade, and it is indicator number 3 for both grades.

Numbering of the Other Performance Indicators

Other performance indicators also utilize four data points but they are formatted slightly differently. Extra Criteria, just like the Oregon performance indicators, utilize 4. EC designating extra criteria, a number indicating the corresponding standard, a number indicating the grade level (8 or 12), and the number of the indicator.

This means that this performance indicator is Extra Criterion. It is from the eighth standard (Advocacy). It is a middle school level indicator, needing to be completed by 8th grade, and it is indicator number 2.

Numbering of National Performance Indicators

National standards from the HECAT are broken into topic area (V-Violence Prevention, SH-Sexual Health, and HBO-Healthy Behavior Outcomes). The topic is directly followed by the standard it falls under (1-8), followed by the grade level, then the indicator number. National Sexuality Benchmark Standards (NSBS) also use topic areas (PD-Strand 2: Puberty and Adolescent Development, ID-Strand 3: Identity, PR-Strand 4: Pregnancy and Reproduction, HR-Strand 6: Healthy Relationships, and PS-Strand 7: Personal Safety). The topic area is followed by a number designating the grade level, then initials corresponding to the standard ((1-8 (CC-Core Concepts, INF-Analyzing Influences, AI-Accessing Information, IC-Interpersonal Communication, DM-Decision Making, GS-Goal Setting, SM-Self Management, and ADV-Advocacy)). These are followed by the indicator number.

That means that this performance indicator is from the national standards, HECAT. It is from the second standard (Analyzing Influences). It is a high school level indicator, needing to be completed by 12th grade, and it is indicator number 6.

This performance indicator is found in both Oregon's Health Education Standards (the first number) as well as the NSBS (the second number). It is from the third standard (Accessing Information). It is a high school level indicator, needing to be completed by 12th grade, and it is indicator number 10 for Oregon and number 1 for the NSBS.

Bold vs. Regular Performance Indicators

Reviewers may notice that some of the performance indicators are bold and others are in regular (or unbolded) font. When a performance indicator is bold, it indicates that it corresponds to Oregon statute or rule. In other words, there is law or policy in Oregon mandating that those performance indicators be taught in schools at the corresponding grade level.

Now that we know more about the criteria within the OHRCATs - the Standards and Performance Indicators, let's take a look at completing the tool using this criteria!

Checking the Boxes

Knowing when to check the boxes can be challenging. This section goes deeper into the Standards and education expectations to help reviewers understand when a performance indicator is met.

Considerations when Checking the Boxes

The OHRCAT development committee developed a set of expectations to help all reviewers complete the curriculum review process in a cohesive and comprehensive way. Some key concepts and resources that provide shared language, definitions, and foundations for all reviewers to start from include:

• Read and be familiar with PEC position papers, including: “Primary Prevention of Sexual Violence,” “Promoting Healthy Sexuality as Sexual Violence Prevention,” and “Intersections of Oppression and Sexual Violence” prior to reviewing curricula with this tool. These can be found on the SATF website at: https://oregonsatf.org/resources-for-prevention;
• and understand that healthy relationships and sexual health promotion are violence prevention.

Considerations when Checking the Boxes

The OHRCAT development committee also identified some review strategies that they request all reviewers utilize in the review process. These include:

• Assume that the people using this curriculum do not have a background in this work, and therefore would not necessarily have the same knowledge to adapt any components of the curriculum to meet a performance indicator;
• and Assume that the curricula are self-contained – nothing will be added or subtracted when delivering the curriculum. Each review reflects the performance indicators met within a curriculum, without additions or changes.

These strategies support a review of each curriculum as a stand alone resource, and allows the review team to provide more directive recommendations and resources to support curriculum implementation in Oregon.

Knowing when to check the boxes

One tool that can be used when thinking about the standards and performance indicators is called Bloom's Taxonomy. This provides us a way to think about different levels of learning which are designated through the use of verbs, which almost all of the performance indicators start with. This pyramid shows the different levels of Bloom's Taxonomy.

The Verbs

Each performance indicator uses a verb (or verbs) to describe what students are supposed to be able to do. These correspond to a level of learning in Bloom's Taxonomy. We can use the next slide to decide which level they are at.

The Verbs: A Cheat Sheet

Creating: compile, compose, construct, create, design, develop, form, imagine, organize, predict, etc.

Evaluating: assess, choose, compare, debate, decide, defend, evaluate, explain importance, justify, Predict, Recommend, re-frame, etc.

Analyzing: analyze, categorize, compare, contrast, differentiate, examine, investigate, question, recognize, research, etc.

Applying: apply, choose, demonstrate, determine, establish, identify, model, practice, prepare, report, show, use, etc.

Understanding: classify, comprehend, define, describe, discuss, empathize, give examples of, identify, recognize, research, summarize, understand, etc.

Remembering: choose, describe, explain, identify, label, list, name, recall, relay, restate, select, summarize, tell, etc.

The Verbs: An Example

Investigate and Assess

Establish

Practice

Using the four performance indicator examples we just highlighted, along with the sample verb lists from the last slide, we can see what level of learning each of those standards falls on.

• Investigate and Assess are both verbs used in the Evaluating level of Bloom's Taxonomy;
• Establish and Practice are both used in the Applying level;
• And Describe is used in both the Understanding and Remembering levels.

Describe

Using the learning level to measure success

Once we know which learning level the standard is working to meet, we are able to measure it against the skills goals for each of those learning levels. Examples of these can be found on the next slide.

Bloom's Taxonomy:

Skills to be demonstrated by students include...

Remembering:

• observation and recall of information

• knowledge of dates, events, places

• knowledge of major ideas

Applying:

• using information effectively

• using methods and concepts in new situations

• solving problems using acquired skills or knowledge

Analyzing:

• seeing patterns

• organization of parts

• recognition of hidden meanings

• identification of components

Understanding:

• the ability to explain ideas or concepts

• translate knowledge into new context

• predict consequences

Creating:

• using old ideas to create new ones

• relating knowledge from several areas

• predicting, drawing conclusions

Evaluating:

• compare ideas

• make choices based on reasoned argument

• verify value of evidence

• recognize subjectivity

Putting this all together...

So, looking back at the examples that we used earlier, we know that Describe is a verb found at the Understanding level of Bloom's Taxonomy. This means that in order for a curriculum to meet this performance indicator, students need to achieve the following:

Understanding:

• the ability to explain ideas or concepts

• translate knowledge into new context

• predict consequences

Therefore, when deciding if this standard is met, it is up to reviewers to decide if students have the ability to explain the intersections of varied identities as a result of this curriculum?

Putting this all together...

Looking at another example that we used earlier, we know that Establish is a verb found at the Applying level of Bloom's Taxonomy. This means that in order for a curriculum to meet this performance indicator, students need to do the following:

Applying:

• using information effectively

• using methods and concepts in new situations

• solving problems using acquired skills or knowledge

Therefore, when deciding if this standard is met, it is up to reviewers to decide if students are able to use personal goal setting skills they have developed and fostered in the curriculum, and apply these skills to decisions around when to engage in sexual activity.

Other Considerations when Checking the Boxes

Many of the performance indicators include multiple variables, like the example listed above. For each performance indicator, the concept and skill (ex. describing intersectionality) as well as the specific variables (ex. gender, race, etc.) are equally important. It is useful to note if there are parts of the performance indicators that go unmet. For example; students are able to describe the intersections of gender, race, and sexual orientation because of the curriculum, but the curriculum doesn't discuss ability or ethnicity at all. If the concept and skill is met, but not all of the variables are met, check the box as well as document these gaps in the notes sections provided throughout the OHRCAT. If a curriculum talks about gender, race, and sexual orientation, but students won't be able to describe the intersections of varied identities, this performance indicator would not be met and the box would not be checked.

When in Doubt...

It's not always easy to decide when to check boxes. That is why SATF's prevention coordinator is available to answer any questions that you may have while reading and reviewing a curriculum! Please don't hesitate to get in touch! Now that we know more about filling out the tool, let's talk briefly about some helpful review strategies.

The Review Process

Review teams are made up of individuals with expertise in sexual health promotion and violence/abuse prevention education. In order to have a diverse range of voices submitting scores and feedback on a curriculum, each curriculum has at least three different reviewers who have been trained in using the OHRCAT and completing the review process. They submit scores, notes, completed accuracy and accessibility charts, and copies of their completed OHRCAT.

Strategies for Reviewing

Different reviewers have employed different strategies for reviewing each curriculum and providing scores as well as notes on each. Some strategies reviewers have identified as supporting them in an effective review process include:

• Reading one or two lessons at a time, then going through the complete OHRCAT to document which performance indicators were met.
• Noting which lessons meet which performance indicators, or which page(s) the lesson and/or activity is on. This can be done in the margins of the OHRCAT.
• When possible, reading through a curriculum once, then reading through it again while documenting the performance indicators met throughout.

Taking Notes

It is important for reviewers to take notes about the curriculum while reviewing it. Some notes might be documented in the accuracy and accessibility charts, which are described in "The Tool" section of this presentation. Some notes may be best documented in the Notes section at the end of each standard, or in the overall Notes section included on the summary score sheet page. These notes help the review team make recommendations and highlight any additions that would need to be included in order to meet more of the performance indicators.

Review Meeting

Once individual reviewers complete their review of a curriculum, the review team will meet to discuss the curriculum, analyze the group's scores, and provide qualitative feedback. This review meeting is where we develop recommendations and highlight the successes in curricula. This meeting is either conducted in person with all review teams, or in a shorter web-based meeting.

There are lots of different strategies for reviewing a curriculum. If you find one that works for you, please let us know, so we can include it in our review strategies! The last step in the review process is what happens with all of the scores and feedback! Let's talk about that for a moment in the last section of this presentation!

Submitting Scores

Once reviewers have completed reviewing a curriculum, they will submit their scores to the Prevention Program Coordinator at the Oregon Attorney General's Sexual Assault Task Force, Meg Foster at megan@oregonsatf.org. This will include scanning and emailing the entire OHRCAT a reviewer completed (Summary score sheet, accuracy and accessibility charts, and the individual standard score sheets).

What happens with the Scores?

SATF's Prevention Program Coordinator will compile the scores to share with the review team for discussion at the review meeting. An average score will be tallied, signifying the total percentage of performance indicators that each curriculum met. This percentage will accompany feedback from the review team in two categories: "Framing" which highlights what the curriculum does well, as well as any recommendations the review team has for reframing components of the curriculum; and "Additions" which includes recommendations for what can be added or included to support that curriculum in meeting more of Oregon's Health Education Standards.

Summary Score Sheet

All of the information listed on the last slide, is added to the Summary Score Sheet which is shared on SATF's website:

https://static1.squarespace.com/static/636973864f6528145b7b9b13/t/63ed136cdd831625446d0037/1676481389808/Curriculum+Review+Summary+Document+3.2021.pdf

What happens to the completed OHRCATs?

Completed OHRCATs are saved at the SATF office, in order to record and document the different performance indicators each curriculum meets. They serve as a resource for educators when they are looking to fill any gaps a curriculum might have in meeting Oregon's health education standards.

Any Questions?

That's it! That is an overview of the curriculum review process. This process has been beneficial in developing a library of reviewed curricula, as well as beneficial for the reviewers as they support and enhance their own programming. We look forward to working with you and supporting you through the next round of curricula reviews! If you have any questions or feedback on this presentation or the OHRCAT Curriculum Review Process, please contact SATF's Prevention Program Coordinator, Meg Foster at megan@oregonsatf.org.