Using-and-interpreting-analogue-designs

Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves materials that approximate or describe reality e.g., written vignettes, vi

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/8027167 Using and interpreting analogue designs

Article  in   Work · February 2005

Source: PubMed

CITATIONS

8

READS 518

2 authors, including:

Some of the authors of this publication are also working on these related projects:

evidence based-practice View project

Core concepts in special education research View project

Bryan G Cook

University of Virginia

138PUBLICATIONS    4,522CITATIONS    

SEE PROFILE

All content following this page was uploaded by Bryan G Cook on 26 May 2018.

The user has requested enhancement of the downloaded file.

IOS Press

Speaking of Research

Using and interpreting analogue designs

Bryan G Cook∗and Phillip D Rumrill, Jr.

Kent State University, Department of Educational Foundations and Special Services, Center for Disability Studies,

405 White Hall, P.O Box 5190, Kent, OH 44242-0001, USA

Tel.: +1 330 672 0579; Fax: +1 330 672 2512; E-mail: bcook1@kent.edu

Abstract Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves

materials that approximate or describe reality (e.g., written vignettes, videotaped exemplars) rather than investigating phenomena

in real-world settings Analogue research often utilizes experimental designs, and it thereby frequently possesses a high degree

of internal validity Analogue research allows investigators to exercise tight control over the implementation of the independent

or treatment variable and over potentially confounding variables, which enables them to isolate the effects of those treatment variables on selected outcome measures However, the simulated nature of analogue research presents an important threat to external validity As such, the generalizability of analogue research to real-life settings and situations may be problematic These and other issues germane to analogue research in vocational rehabilitation are discussed in this article, illustrated with examples from the contemporary literature

Keywords: Research design, analogue research

1 Introduction

Analogue research is defined by the use of materials

that “mirror” or approximate reality In the field of

rehabilitation counseling, analogue studies investigate

individuals’ (e.g., rehabilitation counselors) reactions

to stimuli such as written, fictitious case files;

video-tapes of individuals acting the part of people with

dis-abilities in a specific situations/contexts; role-playing

particular counseling techniques; or other simulations

of reality [1] The simulated nature of analogue

re-search affords advantages (e.g., allows for utilization

of experimental research designs, can be used to

en-hance the internal validity of research) and

disadvan-tages (e.g., reduced external validity) that must be

con-sidered by researchers and research consumers when

designing, conducting, interpreting, and applying

ana-logue research

Analogue research is often used in the context of

experimental designs; as such, the approach typically

∗Corresponding author.

features the high degree of internal validity associated with experiments (see [4]) In the simplest form of

an experimental design, a researcher randomly assigns participants into two groups, then uses one group as

a control – to which no special treatment is given – and the other as the experimental or treatment group – which receives the intervention being examined Be-cause of random assignment to groups (which reduces the likelihood that the groups differ in meaningful ways other than the application of the intervention), differ-ences that exist between the groups on outcome mea-sures can be attributed to the intervention Because experimental designs control for many alternative ex-planations for significant findings, they are said to pos-sess a high degree of internal validity [5] Research using an experimental design may involve real people

in real situations (called “in vivo” research), but it may

sometimes be more desirable or practical to conduct an investigation using a simulation of reality; in that case, the study would be classified as analogue research Consider the example of a rehabilitation researcher who wishes to determine the effects of consumers’

1051-9815/05/$17.002005 – IOS Press and the authors All rights reserved

94 B.G Cook and P.D Rumrill, Jr / Analogue designs

work histories on assessments of employability by state

Vocational Rehabilitation counselors Specifically, she

is interested in determining whether counselors

evalu-ate employability differently for consumers who have

worked in sheltered workshops than for consumers who

have participated in integrated community-based

em-ployment settings She might prepare case

descrip-tions of fictitious consumers who are identical in terms

of age, gender, years of experience in the world of

work, level of education, functional ability, and

disabil-ity type The only descriptive information that would

be varied is type of employment history For half of the

case reports, the consumer is described as someone who

has worked in sheltered workshops for his or her entire

work history For the other half of the reports, the

per-son is described as having worked in various integrated

community-based settings At random, the counselors

participating in the study would receive these two

dif-ferent descriptions of work history (half of the sample

would receive the sheltered workshop profile and half

would receive the community-based settings profile)

They would then evaluate the consumer’s

employabil-ity, and the researcher would compare the ratings

be-tween the two groups in an effort to determine the

im-pact of employment history on employability

assess-ments Because all case reports are identical except in

terms of the type of employment history given for the

consumer, any differences between the two groups on

employability ratings would be attributed to the

con-sumer’s employment history

This example is considered analogue research

be-cause the treatment to which participants are exposed

(different work histories) is fictitious rather than real

(i.e., participants do not rate the employability of actual

people, but rather a written description of a fictitious

person) Also, in this example, an experimental

de-sign is utilized because a treatment (i.e., exposure to a

description of an individual with work experience in a

community-based setting) is randomly applied to

par-ticipants Because of random assignment, any

statis-tically significant differences in mean ratings between

the control (those who rated the employability of the

in-dividual with work experience in sheltered workshops)

and treatment groups can be attributed to the

indepen-dent variable (i.e., type of work history) rather than

to pre-existing differences between the two groups of

participants Conducting analogue research rather than

an “in vivo” experimental design might be considered

when in vivo experimental research is impossible,

im-practical, and/or unethical for reasons including:

1 the existence/availability of participants with the characteristics of interest is insufficient to conduct the desired research,

2 it is unethical to apply the intervention to partici-pants or to withhold it from the control group,

3 the cost of the intervention is prohibitive, and

4 the intervention is too time-consuming to imple-ment

An analogue study might also be used instead of “in vivo” research if the researcher desires greater control

over the treatment or potentially confounding variables

in order to increase the internal validity of his or her research

Analogue research can also be conducted using quasi-experimental designs in which intact groups con-stitute the control and treatment groups instead of us-ing random assignment However, analogue research that utilizes quasi-experimental designs lacks the high level of internal validity associated with true experi-ments [1,5] Using the example introduced previously,

in a quasi-experimental design, counselors who work

in one state office might receive the description of an in-dividual who has worked only in sheltered workshops, and counselors working in another office in a different county might be given the description of the person with experience in integrated work settings Due to the lack of random assignment to groups, it is possible that a difference in mean ratings between the groups is attributable to the different locations in which partici-pants’ offices are housed – rather than to the differences

in the consumers’ work experience as described in the fictitious case reports

2 Strengths of analogue research

In regard to internal validity, analogue research

pro-vides a potential advantage as compared to most “in vivo” research Because of the simulated nature of

analogue research, the researcher can manipulate the two (or more) conditions in experimental (or quasi-experimental) designs in such a way that they are ex-actly the same except as related to the independent or

treatment variable For example, in an “in vivo” study,

a researcher might introduce actual rehabilitation coun-seling clients with different work experiences to re-habilitation counselors for employability assessments However, characteristics other than work history might affect the counselors’ ratings of employability (e.g., age, disability type, level of education) Even if the

researcher was able to match the clients whom

coun-selors rate on several pertinent variables, the client with

experiences only in sheltered workshops might, for

ex-ample, slur his speech, whereas the client with work

experiences in integrated community settings does not

It is possible, then, that differences in employability

ratings would be due to differences in speech rather

than work history In analogue studies, the researcher

has tight control over every aspect of the stimuli

pro-vided to participants and can ensure that the only

dif-ference between the stimulus provided to the control

and treatment groups lies in the independent variable

being investigated (in this case, work history)

Analogue research also gives the researcher

com-plete control over how to present the treatment

condi-tion or independent variable to participants For

exam-ple, let us assume that a researcher wishes to investigate

the impact of how a client’s disability occurred on

reha-bilitation counselors’ attitudes toward the client In an

in vivo study involving actual clients, counselors may

not know or remember the exact cause of disability for

each client, or may have an incomplete understanding

of the cause of the disability In this case, an analogue

study in which a written vignette describes in detail

how the disabilities of two fictitious clients occurred

might constitute a preferable method for examining the

effect of disability attribution on counselors’ attitudes

Two different causes of a disability could be described

(e.g., one client was involved in an accident beyond

his control, the other client’s disability came about due

to reckless driving) and the descriptions randomly

dis-tributed to participants Assuming that all other

char-acteristics are held constant in the vignettes, any

differ-ences in attitudinal ratings between the two groups can

be attributed to the independent variable (i.e., cause of

disability)

3 Limitations of analogue research

An important limitation of analogue research has to

do with external validity, or the degree to which

find-ings generalize beyond the conditions present in the

investigation If a study has weak external validity, the

findings may not pertain to participants or in contexts

that are not exactly like those in the investigation

Be-cause analogue designs, by definition, involve the

ap-proximation of reality, it is highly possible that in

real-life conditions results may differ Referring back to our

example of an analogue study comparing the effects of

different client work histories on counselors’

employ-ability ratings, it is possible that the fictitious case files may be written in such a way that strongly emphasizes work experience Perhaps the ratings regarding em-ployability that counselors give to the vignettes over-estimate the impact of work-experience due to the way the vignettes are written In reality, counselors’ feel-ings regarding the employability of a client might de-pend more on relevant skills, communication, appear-ance, and other factors than on the type or location of work experience

Threats to external validity can be minimized by making the approximation of reality as close as possi-ble In other words, written vignettes, films, or other simulated materials should be as realistic and detailed

as possible However, analogue designs can never truly replicate reality; no matter how realistic a film, written description, or simulation is, it will never truly equate with actually experiencing the situation and the inter-vention Therefore, one can not be sure that the re-sults of analogue research will generalize to real world environments Other strategies for enhancing external validity in analogue research include using large, repre-sentative samples of research participants; replicating previous studies with slight variations in content or ad-ministration of the stimulus; and following up analogue

studies with in vivo experiments.

4 Examples of analogue research in disability literature

Gouvier et al [3] designed a study that used analogue research and an experimental design to investigate the effect of disability type, job complexity, and public con-tact on hiring preferences regarding job applicants with disabilities In the study, 295 undergraduate students taking upper division business courses at a large univer-sity (who represented potential future employers) read fictitious job descriptions and four fictitious resumes of individuals who had closed head injuries (CHI), chronic mental illness (CMI), developmental disabilities (DD), and back injuries (BI) Participants were asked to rate the desirability of each applicant for two jobs: janitor (low complexity) and phone operator (high complex-ity) Finally, participants were randomly assigned to one of two groups One group read job descriptions indicating that the applicant would work during the day (high public contact); the other group believed that the available positions were for night jobs (low public con-tact) On one of the dependent measures utilized by Gouvier et al., participants completed “paired

compar-96 B.G Cook and P.D Rumrill, Jr / Analogue designs

isons between all possible pairs of applicants for each of

the two jobs” (p 178) In other words, participants

se-lected whom they would hire from the six possible

pair-ings (CHI/CMI, CHI/DD, CHI/BI, CMI/DD, CMI/BI,

DD/BI) for each job (low and high complexity)

Results indicated a significant main effect for

dis-ability type and a significant two-way interaction

be-tween disability type and job complexity The

appli-cant with a back injury was more likely to be hired than

applicants with the three other disabling conditions as

a phone operator Likelihood of being hired for this

position did not differ among the applicants with the

other three disabilities However, for the janitor

posi-tion, the applicant with mental illness was significantly

less likely to be hired than the other three applicants–

for whom the likelihood of being hired did not

signifi-cantly differ No significant results were found in

rela-tion to amount of contact with the public The authors

concluded that bias toward applicants with disabilities

varies according to different job requirements The

au-thors also noted that the differences between a real job

application process and the simulated conditions of the

analogue study “may limit the generalizability of the

results” (p 180)

Broussard et al [2] also reported the results of

ana-logue research utilizing an experimental design Their

study investigated the effects of attribution of

respon-sibility for a disability, race of consumer, and race of

rehabilitation counselor on counselors’ predictions

re-garding consumers’ personal adjustment to disability

and vocational outcomes One hundred and fifty (150)

rehabilitation counselor trainees (juniors, seniors, and

graduate students) from two universities (a historically

black Southern university and a predominantly white

Midwestern university) participated in the study Each

participant read a fabricated vocational evaluation

re-port that contained “vocational interest results,

aca-demic aptitude scores, recommendations for next steps,

work history, level of education, and a brief job

de-scription in which the individual was interested” ([2],

p 246) These elements of the vocational evaluation

re-port remained constant, whereas the investigators

var-ied attribution and race

The client was described as exhibiting either

charac-terological self-blame attribution (i.e., “I’m the kind of

person to whom these things always seem to happen I

deserve my fate and have nobody to blame but myself

.it’s just the way it was meant to be” [p 247]) or

be-havioral self-blame (i.e., “I think in the past I’ve taken

too many chances and not considered the consequences

of my actions It seemed like this was going to happen

to me eventually I regret that I was not more careful

and avoided it” [p 247]); and as either Caucasian or African American Thus, four conditions (behavioral attribution – Caucasian consumer, behavioral attribu-tion – African American consumer, characterological attribution – Caucasian consumer, and characterologi-cal attribution – African American consumer) were ran-domly distributed to participants (who themselves var-ied by race – being either Caucasian or African Ameri-can) After reading the fabricated vocational evaluation reports, participants completed a two part instrument on which they predicted the client’s personal adjustment

to disability and vocational outcomes

A significant main effect was observed regarding consumer attribution for prognosis of personal adjust-ment Participants predicted better personal adjust-ment for fabricated clients who made behavioral attri-butions than they did for those making characterolog-ical statements Therefore, the authors conclude that,

“counseling trainees should be made aware of how their own perceptions of attribution, and the attribution self-beliefs of the consumers with whom they work, may influence the decisions they and their consumers make and ultimately affect the success of consumers in the rehabilitation process” (p 253) An additional analy-sis was conducted to determine the effect of sameness

or difference in race between trainee and consumer on participants’ predictions Personal adjustment scores were found to be higher when the consumer and trainee were of the same race

5 Conclusion

Analogue research has been and will continue to be utilized in rehabilitation when researchers want to en-hance the internal validity of their investigations or

con-duct research in situations where in vivo experimental

research is not feasible Analogue studies are of value

to the field because they can investigate the effect of

an intervention or stimulus with a high level of inter-nal validity However, decreased exterinter-nal validity is

a critical concern The simulated nature of analogue research means that research findings may not gener-alize or apply to real-life situations As such, we rec-ommend that researchers use analogue research as an initial inquiry to be followed by additional studies in-volving real people in real situations By understanding the strengths and limitations of analogue research, re-habilitation practitioners are able to appreciate the im-pact that analogue findings can have on field practice, especially as they apply to the clinical judgments that are made concerning clients

[1] J.L Bellini and P.D Rumrill, Research in Rehabilitation

Coun-seling: A Guide to Design, Methodology, and Utilization,

Springfield, IL: CC Thomas, 1999.

[2] S.L Broussard and W Crimando, Effects of consumer

attribu-tions of disability and race on rehabilitation trainees’

predic-tions of rehabilitation outcomes, Rehabilitation Education 16

(2002), 243–254.

[3] W.D Gouvier, S Systma-Jordan and S Maryville, Patterns of discrimination in hiring job applicants with disabilities: The

role of disability type, job complexity, and public contact,

Re-habilitation Psychology 48 (2003), 175–181.

[4] P.P Heppner, D.M Kivlighan and B.E Wampold,

Re-search design in counseling, (2nd ed.), Belmont, CA:

Brooks/Cole/Wadsworth, 1999.

[5] A.E Kazdin, Research Design in Clinical Psychology,

Need-ham Heights, MA: Allyn & Bacon, 1998.