Analysis of Measurement Tool
Assessing kindergarteners’ mathematics problem solving: The development of a cognitive diagnostic test
Cognitive Diagnostic Assessment
Math problem solving (MPS) become important:
1. Student: provide oportunity for student to use content knowledge.
2. Teacher: understand the students ability of math
- Measures specific knowledge structures and processing skills.
- Provides insights into cognitive strengths and weaknesses.
Previous assessment: observations, informal conversations and interviews.
Future
Instrument: Clasical Test Theory (CTT) / The Item Response Theory (IRT):
- Total test score used to represent children's ability or achievement.
- Ranking or set standards.
- Differentiating passing vs. non-passing children.
- creating and define cognitive model related to MPS
- Develop problem solving: realife assessment
Content:
1. Reason develope of CDA
2. Previous research
3. Purpose
4. Structure and component.
5. Constructure process
6. Quality of measurement tools
7. Strengths and weakness
Assessment Tool Analysis
The purpose, structure, and construction process of the measurement tool are analyzed for assessing mathematics problem-solving ability in kindergarteners.
Quality Assessment of Measurement Tool
Strengths
Weaknesses
Item analysis is a critical component of quality assessment in measurement tools, as it helps ensure the validity, reliability, and effectiveness of the items in measuring the intended construct.
- Mixed CDM model: selecting the most appropriate cognitive.
- CDM: optimally use information during diagnosis, and they can obtain the interactions be- tween attributes,
- The quality of the Q-matrix was analysed by using HCI and the psychometric characteristics to create CDT-MPS test.
- Grouping boys and girl, rural and urban.
- Detailed Breakdown: Breaks down math problem-solving skills into clear parts, helping teachers and parents understand children's abilities better.
- Clear Measurement: Shows exactly which skills each test question measures, helping pinpoint why a child succeeds or struggles.
- Hands-on Experience: Provides practical guidance for creating similar tests, offering insights into how to make them effective.
- Starting Challenges: Early attempts faced challenges in fitting the test questions to specific models, showing the importance of getting the model right from the start.
- Method Challenges: While the test showed good results, more research is needed to find better ways to measure skill differences when using multiple models.
- Test Design Issues: Making real-life problems for big tests is tough, so there's still work to do in making these tests reflect everyday situations better.
The Previous Precedure of CDA
1. Construct a cognitive model
Attribute structure: Children knowledge and skills to solve problem.
2. Develop cognitive diagnostic test:
- Reveals test scores, attribute mastery patterns, and probabilities.
- Mastery patterns indicate mastered attributes.
- Mastery probabilities show degree of mastery.
3. Literature review:
- Define cognitife attribute
- Structure of cognitive attribute
Cognitive attribute and structure
4. Test Quality of Cognitive model
- Developed an initial cognitive diagnostic test.
- Recruited 627 children from Shanghai.
- Examined the quality of the cognitive model.
Results:
- Average Hierarchy Consistency Index (HCI) of 11 cognitive attributes was .749.
- Indicates good fitting between the cognitive model and children's response data.
- Suggests the cognitive model and Q-matrix of the test were effective.
Intended Purpose of the Tool (CDA)
1. Assessing Kindergarteners' Mathematics Problem Solving (MPS):
- Evaluate specific cognitive attributes and mastery patterns related to problem-solving abilities in numbers and operations.
- Use a 0-1 scoring system to show mastery of cognitive attributes, giving a detailed assessment of children's problem-solving skills.
0: Correct response
1: Incorrect response
2. Contribution to Understanding How Kids Learn Math:
- Offer a thorough evaluation of cognitive strengths and weaknesses in math problem-solving.
- Analyze mastery patterns and probabilities of cognitive attributes to understand individual children's mastery levels and areas for improvement.
3. Helping Teachers Teach Better:
- Provide teachers with insights into how kids think about math to enhance teaching methods.
- Simplify the process for teachers to support children who need extra help in math.
The structure and componen of the measurement tool
- Cognitive Model: The theoretical framework that outlines the knowledge and skills needed for solving math problems.
- Cognitive Attributes: Specific aspects of mathematical thinking the test aims to assess (e.g., counting, addition).
- Q-Matrix: A blueprint that links test items to the targeted cognitive attributes.
- Test Development: The process of creating test items based on the Q-matrix and cognitive model.
- Assessment Components: The different outputs of the test, including scores, mastery patterns, and probabilities, which provide detailed insights into a child's mathematical problem-solving abilities.
The cognitive attributes of Kindergartners's MPS
The process of developing a cognitive diagnostic assessment (CDA)
1. Q-MATRIX CONSTRUCTION
Steps in constructing a Q-Matrix for assessing cognitive attributes in kindergarteners.
REACHABILITY MATRIX
Represents direct and indirect relationships among attributes.
LATENT KNOWLEDGE STATES
Reduced to eliminate irrational patterns based on attribute hierarchy.
FINAL MEASURE PATTERNS SELECTION
Removed infeasible or redundant measure patterns, ensured feasibility and understandability for kindergarteners, considered test time constraint of 30 minutes.
2. the development of cognitive diagnostic test
Construction of Test Items:
a). Measure patterns from the refined Q-matrix were established first.
b). Applied problems incorporated semantic construction, mathematical relations, numerical skills, and problem-solving strategies.
Defining Cognitive Attributes
a). The research team defined the specific cognitive attributes necessary.
b). The items were adjusted to align with the defined cognitive attributes and the measure patterns in the Q-matrix.
c). This process was repeated until a consensus was reached.
Consider the difference in life experiences between urban and rural children.
Scoring criteria: these were aligned with the cognitive attributes measured by each item
3. validation procedures
EXPANSION OF Q-MATRIX
The initial Q-matrix consisted of 25 measure patterns representing different cognitive attributes, then 13 items were added based on investigation.
SCORING SYSTEM
HIERARCHY CONSISTENCY INDEX
The CDT-MPS test used a score system of 0-1, where 1 indicates attribute mastery and 0 indicates lack of mastery.
The obtained HCI value of 0.73 signifies good consistency between observed and expected response patterns from attribute hierarchy.