Assessment in counseling chapter 4

∗ Concerns what instrument measures and how well it does so ∗ Not something instrument “has” or “does not have” ∗ Informs counselor when it is appropriate to use instrument and what ca

Validity and Item

Analysis

Chapter 4

∗ Concerns what instrument measures and how well

it does so

∗ Not something instrument “has” or “does not

have”

∗ Informs counselor when it is appropriate to use

instrument and what can be inferred from results

∗ Reliability is a prerequisite for validity

Validity

∗ Evidence Based on Test Content

∗ Evidence Based on Response Processes

∗ Evidence Based on Internal Structure

∗ Evidence Based on Relations to Other Variables

∗ Evidence Based on Consequences of Testing

Validation Evidence: Standards for

Educational and Psychological Testing

∗ Degree to which the evidence indicates that items, questions, or tasks adequately represent intended behavior domain

∗ Central focus is typically on how the instrument’s content was determined

∗ Content-related validation evidence should not be confused with face validity

Evidence Based on Test Content

∗ Concerns whether individuals respond in a manner

consistent with construct measured

∗ May also examine information processing differences

by subgroup

∗ Computer simulated performances

Evidence Based on Response Processes

∗ Examining internal structure using factor analysis

∗ Can also examine internal structure of instrument for

different subgroups (differential item functioning)

Evidence Based on Internal Structure

∗ Correlational method

∗ Convergent and discriminant evidence

∗ Multitrait-multimethod matrix

∗ Prediction or instrument-criterion relationship

∗ Concurrent and predictive validity

∗ Concurrent validity

∗ Predictive validity

Prediction/Instrument-Criterion

Relationship

a = y-intercept

b = slope

X = score

xy S

2

=

Standard Error of Estimate

∗ The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity

Decision Theory: Expectancy Table

Expectancy Table (cont.)

Expectancy Table (cont.)

∗ Method of combining validation studies to determine if validity evidence can be generalized

∗ Must have a substantial number of studies

∗ Must use meta-analytic procedures

Validity Generalization

∗ Examples of social consequences:

∗ Group differences on tests used for employment selection

∗ Group differences in placement in special education

∗ Counselors should consider both validation evidence and social implications

Evidence-Based Consequences

of Testing

∗ Gradual accumulation of evidence

∗ Counselor must evaluate the information to

determine if it appropriate for which client under

∗ Examining and evaluating each item in instrument

∗ Focus is on each item and establishing items that measure ability or level of a latent trait

∗ Involves examining item characteristic curve and calibration of each individual item

Item Characteristic Curve