Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

EDUC 532: Assignment 3

Qualitative Coding & Interviews
by

Kristen S

on 24 November 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of EDUC 532: Assignment 3

Organizational Learning Project
Assignment 3
EDUC 532

Kristen Shultz

University of Southern California
Dr. Holly Ferguson

The purpose of this project is to understand
how data is used
at a specific organization - ABC Elementary School - by site administrators. In this study,the types of data, uses of data, and effectiveness of data are revealed through
qualitative research
- interviews. Moreover, through these interviews, the idea of
changing
how data could be used emerged as a prominent hope for the organization.
1. Purpose
"ABC Elementary School"
- 1 school school district in Imperial County, CA
- 2011 Base API: 761
- 359 Students in K-8 school
- 328 identify as Hispanic/Latino
- 22 identify as White
- 339 are socioeconomically disadvantaged
- 251 are designated as English Learners
- For the 2012-2013 school year, the school was placed in Year 5 of Program Improvement
- The school did not make its AYP for the 2012-2013 school year based on the performance of the school's subgroups on the annual CST
-
Data documents analyzed later support this information.


Information and Data from: Data Quest
2. Sample and Setting
Eight open-ended questions based on the guiding researching question - how do employees are your organization use data? - were appropriated and revised from a thorough list created in class by the cohort.

Using each other as peer scaffolds to critique and revise the list, a pool of questions was created from which to select or guide our own interviews.
3. Interview Protocol
The interviews were conducting over the phone on April 10 and April 11, 2013. The questions were already prepared in a word document, with space left between each question so that I could take notes on my laptop and record their responses.

Mrs. X's interview was held on Wednesday, April 10 from 6:12pm to 7:15pm.

Ms. Y's interview was held on Thursday, April 11 from 1:00pm to 1:29pm.
4. Interview Procedure
The Creswell Model

1. Organize & prepare the data for analysis.
2. Read through all the data.
3. Code the data.
4. Generate description.
5. Create narrative.
6. Interpret the data.
5. Analysis
and
6. Findings (Part 1)

1. Conclusions: A synthesis of the findings
2. Further inquiry questions
Findings (Part 2)
The Guiding Research Question
Sample: Who was Interviewed?
Superintendent/Principal
and
Assistant Principal/Categorical Programs Coordinator
were selected for the interviews
Rationale:
- As with any qualitative inquiry, participants are
purposefully selected.
Each respondent is sensitive and knowledgeable of their environment and of their role at the school (Creswell, 2009).
- The Superintendent/Principal is in her first year at the school and region but not her first year in an admin position. The Assistant Principal was has been at the school since 2006 but has taken a more data-related role since Mrs. X started at the school in the fall. Ms. Y is also a native to the area and knows the culture well.
How do employees at your organization use data?

This question is rooted in qualitative data and inquiry practices. Here we are using qualitative research to recognize the strengths and limits of existing data practices as well as look to the future for evidence-based solutions (Creswell, 2009).
All names, organizations, and locations are identified with pseudonyms but all case study setting related and interview data are real.
Why these 8 questions and these probes?
Why this site?
- Ongoing research and data analysis relationship with this school as an educational consultant
- Work primarily with upper grade teachers (grades 6-8) to institute new Language Arts curriculum
- School has made gains in its API and its AYP with many of its subgroups but still remains in Program Improvement
- School has new administrator who was hired for "radical" change according to her Board President
- School is looking to close the achievement gap and prepare for the Common Core Standards through enrichment curriculum, project-based learning, and an emphasis on writing
- Despite ongoing consulting relationship, the concept of data has been loosely defined and discussed. This begs the questions:
how do employees use data? And, what do they consider to be the definition and purpose of data?
Since I have a working relationship with the school but have little understanding of what, how, and why data is used according to the administrators, these questions provide a rich and comprehensive picture of data from the perspective of those who make decisions based on the data.

Each respondent was asked the same set of scripted questions and probes.
Note Taking
There are many pros and cons to note taking as a recording strategy. While it is convenient and economical, it means the researcher must both generate and record data at the same time, make decisions about what to record and what to leave out, and can take away from the 'in the moment' feel of an interview.

In order to take notes, the interviewer must write down key phrases or terms, lists of critical points, and anything else that "captures" the respondent's point of view (Patton, 1987, p. 139).

Furthermore, if transcription is not being used to facilitate the data collection process, these notes must be thorough so as to include actual and accurate quotations (Patton, 1987).
Phone Interviews
While phone interviews are not ideal and create some limitations as the interviewer is unable to see non-verbal cues or see how the interviewee is behaving during the interview, it was the most convenient method given the time period and distance between myself and the school (Patton, 1987).

Ideally, qualitative researchers collect data in the field - in natural settings where they can observe and experience the setting at hand and have face-to-face interaction with the subjects (Creswell, 2009).

However, despite this, the purpose of this qualitative interview-to gain insight into the organization, learn how people view data and capture an individual's perceptions-was maintained and upheld despite being unable to read the interviewee visually (Patton, 1987).
All Documents can be made available in a more readable size via email.
...bringing sexy back...
EDUC 532 called. They're bringing sexy back.
“And teachers are not prepared to find these alternative methods – they aren’t gaining the experience to look at data, know what it means, and use it to change and improve their practices instead of just re-teaching.”
“Use multiple kinds of data and multiple methods or style to guide instruction…but need more checkpoints along the way to show depth of knowledge and retention of a concept but it’s difficult to determine since there are flaws with data and how teachers are currently using data.”
“[Teachers need to] try different models of instruction…”
“…help [teachers] see choices like reteaching, lessen the point value if all the kids failure, help decide what reliable data or the assessment is…”
Theme 6:
Data should point to alternative methods of instruction should the first modality prove unsuccessful for student learning.
“That’s the flaw in the data: it’s great for the moment…”
Data “doesn’t give a good picture of what they know. Can someone come up with a better plan?”
There are “inaccuracies in the data —what if the kid guesses that day?”
“Data is so flawed because you have to ask yourself: what are you assessing?
“The biggest detriment is that it doesn’t help the teachers figure out how to teach [the material] better”
“If I’m looking at CELDT scores, and we have group of students who scored Proficient on the CST, the data doesn’t match up so it causes me to wonder: what is going on?”
“So I have questions about the CELDT: is it reliable?”
There is a disadvantage with data if “you focus too much on the data and not on who it is or who it represents.”
Theme 3:
Data can be unreliable or flawed
“But some people aren’t using it enough”
“Some teachers drag their feet to use [the data program Illuminate]
A barrier to data use is… “time, willingness to use technology, willingness to try out new ways to teaching”
“I would say 50% use [data] all the time, 30% use it occasionally, and 20% don’t use it unless they need to turn something in”
“…might not have been well trained so don’t understand its greatness.”
Teachers are “not skilled in it so don’t value it”
“Even though some teachers will fight me on this…”
There is “some blame going with just looking at data…so there is a lot of finger shaking”
“There was some resistance from the teachers”
“Some teachers take it personally”
“For the most part teachers use data but there are one or two who go through the motions and won’t follow through…”
Theme 2:
Teacher attitudes and knowledge of data can hinder its use in the school or classroom.
What follows includes:

- Quotations that support each theme (part 1). Each theme is color coded to match the highlighted sections of the coded interview documents.

- Conclusions: A synthesis of the findings based on the narrative popplet, the evidence, and supporting documents (part 2)

- Further inquiry questions (part 2)
Step 6: Interpret the Data
In this step, the researcher develops categories and themes for analysis. Themes are ones that run throughout each of the documents used for analysis, including – the interviews and the data samples.

In a larger study, these themes would be used for different headings in the findings section (Creswell, 2009). However, for the purpose of this project, the themes help to categorize the ways in which data is used and understood by the organization.

Moreover, the descriptions and themes point to an interconnected, rich picture of the setting, people, and situations. While some of the themes stand alone as their own distinct idea, many of the themes were generated out of interrelated ideas concerning how data is used and perceived by the respondents.
4. Generate the Description
With this step, it is necessary to get a “general sense” of the data (Creswell, 2009, p. 185).
- What ideas are the participants saying?
- What is the tone of the ideas?
- What is the impression of the depth, credibility, and use of information?
2. Read Through All the Data
Qualitative data analysis is an ongoing and reflective process that occurs while the data is being gathered. As the researcher, it is important that I collect all necessary data, transcribe the interviews, include any field notes, and sort the data by types (Creswell, 2009).

Moreover, qualitative researchers gather data from across sources in order to triangulate the data and cross-check the consistency of the information (Patton, 2002).
1. Organize and Prepare the Data for Analysis
“In the past, they used the Academic Coach to collect, analyze, and interpret data, but that’s not the coach’s job…”
“…more training and more use of the Academic Coach in the classroom and with the teachers and staff to help them understand the use of data and the advantages of data...having a data expert on my staff to work with the certificated coach would be in my perfect world.”
“Bring back an Academic Coach – they are very valuable.”
“A Coach removes [the stigma of evaluation] – they are friendly, approachable, can give suggestions without offending…and they can use their outside time to learn more…”
Theme 8:
An Academic Coach can help solve data and instruction problems in the classroom or school
“Teachers have access to their own data…in theory, the can recognize their own strengths and weaknesses as teachers.”
“If we see or don’t see a steady climb in student scores, that tells me the teacher’s strengths and weaknesses and where we should place them like using core classes using a multiple subject credentialed teacher - let the teacher shine in the area of focus that they love and shine in…so now every child gets the strong teacher…”
“The lower grade levels became used to meeting with the Academic Coach, being observed, and collaborating on any strengths and weaknesses”
Theme 7:
Data can show a teacher or grade level’s strengths and weaknesses
“Different grades and teachers have different methods – some work as teams to use and analyze data.”
“Teachers have access to their own data…but when they meet as grade teams, they can look across the grade level…”
I turned the process of analysis “over to the teachers after viewing their processes and their models.”
I “use teacher leaders to demonstrate how they use data…”
“Staff is trained and use data during grade level meetings, staff meetings, and professional development.”
“As a former Reading First school, collaboration and teamwork are strong at the primary grades…”
“There is more collaboration because they brought that mentality with them…”
Theme 4:
Data analysis for instruction is largely teacher driven and developed through collaborative efforts.
“…a snapshot of where kids are”
“Shows that moment in time and not what or how they retain information”
“…look across grade level to see how other students in that grade are performing compared to their own”
“I can attach names and faces to [data]”
“…find data on subgroups”
“It helps knowing the students and being in a small school”
“…and really bringing purpose to the test —especially at the upper grades as the students move on to high school”
“…how will they perform on the CST?"
Theme 1:
Data is used to gather information regarding students and student academic performance
A narrative can be used to convey the findings of the analysis - such as a detailed look at several themes as well as:
- Subthemes and interconnected themes
- Specific illustrations
- Multiple perspectives
- Quotations
- Use of “visuals, figures, and tables”
- A “process model”


(Creswell, 2009, p. 189)



A narrative using Popplet…
Step 5: Create the Narrative
Emerging Themes
1. Data is used to gather information regarding students and student academic performance

2. Teacher attitudes and knowledge of data can hinder its use in the school or classroom.

3. Data can be unreliable or flawed.

4. Data analysis for instruction is largely teacher driven and developed through collaborative efforts.

5. Data drives instruction.

6. Data should point to alternative methods of instruction should the first modality prove unsuccessful for student learning.

7. Data can show a teacher or grade level’s strengths and weaknesses.

8. An Academic Coach can help solve data and instruction problems in the classroom or school.

9. Data should include different measures.
4. Generate the Description
Types of data

Uses for data and for whom/by whom?

Role(s) of teachers

Data in relation to instruction

Downfall of data (biases, reliability)

Multiple forms of data and assessment

Data as part of the picture

 
Emerging
Topics
In this step, the research “chunks or segments” the text before bringing meaning to the information (Creswell, 2009, p. 186). Based on the general ideas, I was able to segment the information into some initial themes. I made a list of several topics and then used highlighters to find evidence of these topics in each interview and supporting document.
3. Code the Data
2. Read Through All the Data
Creswell, J. W. (2009). Research design: Qualitative,
quantitative, and mixed methods approaches. Thousand Oaks, CA: SAGE Publications, Inc.
The Creswell Method Applied
“[We need to] use multiple kinds of data and multiple methods and styles (such as instantaneous feedback) to guide instruction…”
“[We should] use different measures – multiple measures…which is hard because accountability is all based on the CST even through you are looking at subgroups.”
Theme 9:
Data should include different measures
“We would be better served by a computer program that would either have students move forward if they got the right answer and prove they understand or re-teach using a different modality if they got it wrong…”
“We need to show it in a whole new way…something data doesn’t show you how to do. And teachers are not prepared to find these alternative methods – they aren’t gaining the experience to look at data, know what it means, and use it to change and improve their practices instead of just re-teaching…”
“[The use of data] should be about how the lessons influence assessment and understanding…either informally or formally assessing students at that moment to check for understanding and gain quick feedback and then influence the instruction.”
Theme 5:
Data drives instruction
In the final step, the researcher is asked to make sense of the data and uncover the lessons learned.

Conclusions are evidence-based and include quotations, theory, and/or related literature. Following the findings, the researcher suggests how the findings may either confirm or diverge from known information (Creswell, 2009).

In essence, this step allows the researcher to synthesize the findings and pose further inquiry questions.
Step 6: Interpret the Data
Based on the general ideas from step 2 and the emerging topics, the data was hand coded using different colors to denote the various topics.
From this, themes and specific descriptions were categorized and defined.
3. Code the Data
General Ideas
2. Read Through All the Data
Further Inquiry Questions
1. If teacher attitudes and knowledge are a concern, what steps are in place in order to improve data literacy for the teachers?

2. Both Mrs. X and Ms. Y espoused the need for an Academic Coach, if that is not within the school's budget, then how does the school and administration plan to work within the budget in order to get the data and instructional support they feel they need?

3.How do data about teachers, the budget, and school plans influence instructional decisions?

4. How will the use of data shift as the school implements the Common Core Standards?

5. How can the administration encourage collaboration for data analysis across all the grade levels?

6. What additional data will provide a more thorough and accurate depiction of individual learners?
Conclusions: A Synthesis of the findings
Popplet Link
Appendix: Samples from Documents
CST Data (Data Quest)
Sample Supporting Data Documents
Data from Data Quest & Ed Data
http://popplet.com/app/#/959131
Not to be confused with popples...
- Data about a student provides a snapshot image of a particular learner in that moment, but it in no way accurately reflects the students' true abilities and knowledge (Themes 1, 3 and data documents)

- Teachers as data experts is a tenuous and loosely defined role because teachers may lack the knowledge, skills, motivation, and efficacy to make informed decisions about instruction based on that data (Themes 2, 4, 5, & 7)

- In an ideal world, multiple types of data drive instruction and help administrators, Academic Coaches, and teachers to make informed decisions to improve student performance through multiple modes of instruction (Themes 5, 6, 8, & 9)

- The building of teacher knowledge about data types, purposes, and uses through collaborative means can improve a teacher's practices and a student's performance (Themes 2, 4, 5, & 6)
Data documents such as those from Ed Data and Data Quest support this theme and show that there is a large body of quantitative data that illustrates student performance and student profiles at a glance.
Making Data Sexy!
"It is a capital mistake to theorize before one has data" - Sir Arthur Conan Doyle, Sr.
Creswell, J. W. (2009). Research design: Qualitative,
quantitative, and mixed methods approaches. Thousand Oaks, CA: SAGE Publications, Inc.

Kurpius, S. R. & Stafford, M. E. (2006). Testing and measurement: A user-friendly guide. Thousand Oaks, CA: SAGE Publications, Inc.

Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: SAGE Publications, Inc.

Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: SAGE Publications, Inc.
For the qualitative process, data for open-ended interviews are quotations (Patton, 1987).
The 8 Questions Selected
Copy/Paste the link to see the complete Popplet
Here it is evident that data is used for the purpose of gathering information on student academic performance. Both respondents commented on how the data shows a moment in time but not the whole picture. However, since ABC Elementary School is relatively small, it is easy to attach the numbers and percents from tests like the CST or the CELDT to individual students.Although the respondents pointed out different kinds of data being used - Mrs. X brought up the student data system Illuminate while Ms. Y brought up the CST, API, and AYP - data is defined as that which points to student performance. This data, moreover, reflects upon the school and not just on that student performance as Ms. Y referred to the school's API score as an indicator of overall student performance. Finally, the data documents support that the data about a school's CST scores shows an overall "snapshot" of academic performance, but until individual scores are analyzed and connections to individual students are made, a snapshot is all the data will provide.
Both Mrs. X and Ms. Y lamented that data is source of conflict between administration and the teachers. Many teachers feel personally blamed for low student test scores, and when administration wishes to improve student test scores, it becomes an attack on that teacher or grade level. This malcontent may stem from a lack of knowledge or skills on the part of the teacher or a lack of mutual understanding of the purpose and use of data between administration and faculty. Alternatively, it may stem from those feelings of blame as teachers feel "personally" attacked when data of poor student performance is discussed during staff meetings, grade level meetings, and Professional Development. In a school that is in year 5 of Program Improvement and with an API score under 800, the added pressure to improve at whatever cost has created a culture of urgency and even blame that is felt and experienced by all individuals. However, both respondents did not describe how they could be part of the solution. Instead, as described in later themes, they believed that outside sources such as Professional Development, a computer system, and/or an Academic Coach could better improve data literacy and teacher attitudes.
Each of the respondents talked at length of the disadvantages of using data such as the CST or programs such as Illuminate to gather information on student performance. Just as data does point to student performance, it does not and cannot provide you a complete picture of that particular student. Both Mrs. X and Ms. Y described the use of quantitative data such as student test scores but these numbers, percentages, and percentiles do not show you what is happening with that student in that moment, how much they have learned, what they really know, or how a teacher can use the data to improve instructional practices. Yet as education has focused on the data provided by CST, the CELDT, and other assessments, that has become the definition of student performance and student aptitude. Additionally, both respondents discussed the flaws in data but did not discuss how they overcome these flaws save for examining more data. Ms. Y did indicate that knowing the students has helped her to overcome some issues of reliability and validity; however, there is little action she can take to rectify these issues. For example, scores may not match up or truly represent a student's ability and questions may be raise, but what can Ms. Y do about it when this is state reported data and state mandated tests?
Based on the responses of both Mrs. X and Ms. Y, there seems to be little formal data analysis system or process set in place for the teachers at ABC Elementary School. Teachers meet during grade level meetings, staff meetings, and Professional Development to discuss data in a collaborative manner, but neither respondent spoke of the processes or protocol for discussing data and its implications. There is a disconnect between how the various grade levels function. The lower grades worked within a formal system when the school had Reading First; however, the upper grades do not have the same knowledge or skills. The transference of that knowledge to the upper grade teachers has been left in the hands of the teachers themselves, and administration has not played a significant role. At the same time, it is the belief of Mrs. X that teacher leaders should play a critical role as peer scaffolds and mentors when it comes to instruction. She acts as a facilitator of the process - a guide rather than a dispenser of knowledge. Nonetheless, the terms "collaboration" and "teamwork" arose throughout both interviews, indicating that the concepts are central to ABC Elementary School's mentality when it comes to data.
The idea that data influences instruction appears in many of the themes and quotations from the interviews. Data is the jumping off point for how and what needs to be taught - more so than the standards. Data, according to the respondents, shows what students, in that moment, know or don't know. It can provide quick feedback and a check for understanding concerning knowledge and skills. While it may not show the whole picture, as described previously, data can point to areas in need of instructional focus. However, at the same time, this data cannot tell a teacher how to teach the material. Mrs. X spoke at length of her concern that data points to what students don't know but does not show a teacher how to re-teach or find alternative methods of instruction. Therefore, those of holes of knowledge may grow as teachers fail to change and improve their own practices. Using and applying data is a process, not simply a number or an end point; instead, it represents a a reflective and active pathway toward better student performance.
Just as data drives instruction, it should point to alternative ways of instruction. Both Mrs. X and Ms. Y emphatically discussed the need for multiple instructional modalities that emerge following the analysis of student performance data. Ms. Y talked about how in her role as Assistant Principal and informal Academic Coach, she often helps teachers find solutions such as alternative methods of instruction when data reveals low student understanding. Mrs. X spoke of the need for students to gain deeper knowledge, not just surface knowledge often presented in initial lessons. Frequent checks for understanding and quick data analysis should point to where students need that deeper knowledge. However, Mrs. X also contends that teachers are not prepared in their own education programs to find alternative methods not described in teacher manuals or the like. Therefore, while data drives instruction and should point to additional instructional methods, it also can indicate a teacher's own strengths and weaknesses.
In some ways, this theme represents a key solution for how employees at ABC Elementary School can and should use data. By using multiple measures and types of data, a teacher or administrator can gain a fuller picture of an individual student, have a more accurate understanding of a student's ability, can look for alternative methods of instruction, and can use other teacher's as resources as they come together with multiple sources of information regarding student performance. Furthermore, with multiple sources of information, a teacher is able to triangulate the data and truly recognize his or her own strengths and weaknesses as an instructor; this lessens the chance for blame as teachers have multiple and hopefully reliable sources of data that show what is both working and not working in the classroom.
Both Mrs. X and Ms. Y spoke of the need for an Academic Coach as someone who can make informed instructional decisions based on student performance data. The former Instructional Coach left the school at the beginning of the academic year with a better offer of employment arose following significant budget cuts at ABC Elementary School. It was Mrs. X's best financial interest to let the Academic Coach go and find outside Professional Development and training opportunities for her staff. However, the absence of that constant presence of the Academic Coach has, according to Ms. Y, affected the efficacy of the teachers and has, she indicated, perhaps led to these feelings of blame. However there is a disconnect between how Mrs. X and Ms. Y view the role of the coach. Mrs. X maintains that is not the role of the Academic Coach to work with data but to work with teachers. Ms. Y feels the two are inextricably related-that data drives instruction and the coach's role is to use that data to help teachers find those alternative methods of instruction and work with teachers in a collaborate manner to improve student performance. From previous conversations, I know that Mrs. X has never worked at a school with an Academic Coach while Ms. Y feels that the coach is an integral member of the staff. Mrs. X has no intention of bringing back an Academic Coach, and instead plans on increasing training and bringing in more outside support. Perhaps these different experiences have lent themselves to different understandings of the purpose, use, and importance, of an Academic Coach.
Implications
The research findings based on the interviews and data documents show that improved data literacy beginning with the very definition of data and its purpose at ABC Elementary School needs to be fleshed out and understood by all stakeholders. While it may be a universal truth that data provides merely a snapshot image of a learner and can be flawed or unreliable, not using data is not an option for any school. Therefore, in order to overcome issues of blame or examining data to find teacher weaknesses, data should and needs to be used to guide and drive instruction, as discussed by both respondents. However, each respondent did not pinpoint detailed ways in which they could facilitate the process of improving instruction based on data. The division between administration and teachers was clear as discussions of teacher blame and resistance surrounding data was brought up. Consequently, a collaborate effort to unpack the meaning of data - at both the denotative and connotative levels - should be chief amongst ABC Elementary School's actions. Finally, the interviews point to a more generalizable issue at hand: the role and use (or mis/overuse) of data in education as basis for accountability. While eliminating all forms of data used at ABC Elementary School is not the solution, a significant overhaul may improve data literacy and attitudes.
Here is where student performance data influences decisions made at ABC Elementary School. If there is not a "steady climb" in student scores, then there is a correlation between instruction and student performance. As Mrs. X was hired to create radical change at the school and improve the school's API score, Mrs. X feels no qualms about making decisions about teacher placement at particular grade levels should those teachers' scores reflect a lack of content knowledge. At the same time, administrators using data in such a way also lends itself to teachers feeling blame and perhaps resisting the use of data. However, Ms. Y counters that the Academic Coach removed a degree of that blame by working with the teachers to improve their instructional practices should data illuminate any weaknesses in the classroom. Thus, there is a difference of opinion concern how to approach teachers when data shows less than desirable outcomes.
Full transcript