No matter how carefully you plan your study, problems almost inevitably crop up when you begin to execute it. Two methods you can use to minimize these problems and ensure the usefulness of the data you collect are conducting a pilot study and adding manipulation checks.
Conducting a Pilot Study
A pilot study is a small-scale version of a study used to establish procedures, materials, and parameters to be used in the full study. Frequently, it is a study that began life as a serious piece of research but “went wrong” somewhere along the way. Th e decorti- cate rat study became a pilot study for this reason. However, many pilot studies are designed from the ground up as pilot studies, intended to provide useful information that can be used when the “real” study gets under way.
Pilot studies can save tremendous amounts of time and money if done properly.
Perhaps you intend to conduct a large study involving several hundred participants in order to determine which of two methods of teaching works best in introductory psychology. As part of the study, you intend to hand out a large questionnaire to the students in several introductory psychology classes. Conducting a small pilot study (in which you hand out the questionnaire to students in only a couple of classes) may turn up inadequacies in your formulation of questions, inadequacies that lead to confusion or misinterpretation. Finding these problems before you train instructors in the two teaching methods, have them teach a full term, and then collect the question- naires from 2,000 students is certainly preferable to fi nding the problems afterward.
Pilot studies can help you clarify instructions, determine appropriate levels of independent variables (to avoid range eff ects), determine the reliability and validity of your observational methods, and work the bugs out of your procedures. Th ey also can give you practice in conducting your study so that you make fewer mistakes when you
“do it for real.” For these reasons, pilot studies are often valuable.
You should also be aware of some negative aspects of pilot studies. Pilot studies require time to conduct (even if less than that of the formal study) and may entail some expenditure of supplies. Where animals are involved, their use for pilot work may be questioned by the local animal care and use committee (particularly if the procedures involve surgery, stressful stimulation, or deprivation). In these cases, you may want to use the best available information to determine procedures and to try to “get it right”
the fi rst time around. Th en only if you guess wrong will the study become a pilot study.
bor35457_ch05_123-157.indd 154
bor35457_ch05_123-157.indd 154 4/22/13 2:52 PM4/22/13 2:52 PM
SUMMARY 155
Adding Manipulation Checks
In addition to the dependent measures of the behavior under study, you should include manipulation checks. A manipulation check simply tests whether or not your inde- pendent variables had the intended eff ects on your participants. Th ey allow you to determine if the participants in your study perceived your experiment in the manner in which you intended. For example, if you were investigating the impact of a person’s attractiveness on how his or her work is evaluated, you might have participants evalu- ate an essay attributed to either an attractive or unattractive author. Th is could be done by attaching a photograph of an attractive or unattractive person to an author profi le accompanying the essay. As a manipulation check, you could have participants rate the attractiveness of the author on a rating scale.
Manipulation checks also provide you with information that may be useful later when attempting to interpret your data. If your experiment yielded results you did not expect, it may be that participants interpreted your independent variable diff erently from the way you thought they would. Without manipulation checks, you may not be able to properly interpret surprising eff ects. Manipulation checks may permit you to determine why an independent variable failed to produce an eff ect. Perhaps you did not eff ectively manipulate your independent variable. Again, manipulation checks provide information on this.
A set of measures closely related to manipulation checks are those asking par- ticipants to report their perceptions of the entire experiment. Factors to be evaluated might include their perceptions of the experimenter, what they believed to be the true purpose of the experiment, the impact of any deception, and any other factors you think are important. Like manipulation checks, these measures help you interpret your results and establish the generality of your data. If you fi nd that participants perceived your experiment as you intended, you are in a better position to argue that your results are valid and perhaps apply beyond the laboratory.
QUESTIONS TO PONDER
1. What is a pilot study, and why should you conduct one?
2. What are manipulation checks, and why should you include them in your study?
SUMMARY
In contrast to casual, everyday observations, scientifi c observations are systematic.
Systematic observation involves making decisions about what, how, and when to make observations. Observations of behavior are made under controlled conditions using operational defi nitions of the variables of interest.
When choosing variables for your study, you should be guided in your choice by research tradition in the area of study, theory, the availability of new techniques or measures, and the limits imposed by the equipment available to you. In addition, you need to be concerned with the characteristics of the measure, including its reliability,
156 CHAPTER 5 . Making Systematic Observations
its validity, and the level of measurement it represents. A measure is reliable to the extent that repeated measurements of the same quantity yield similar values. For mea- sures of physical variables, reliability is indexed by the precision of the measure, and for population estimates, by the margin of error. Th e reliability of the judgments of mul- tiple observers is indexed by a statistical measure of interrater reliability. Th e reliabil- ity of psychological tests can be determined in a variety of ways, yielding test–retest, parallel-forms, or split-half reliabilities. A measure is accurate if the numbers it yields agree with a known standard. Accuracy is not a characteristic of most psychological measures because there are no agreed-upon standards for them. A measure is valid to the extent that it measures what it is intended to measure. Several types of validity assessment exist for psychological tests, including face validity, content validity, con- struct validity, and criterion-related validity. Th e latter takes two forms, called concur- rent validity and predictive validity.
One aspect of systematic observation is developing dependent measures. Your data can be scaled along one of four scales of measurement: nominal, ordinal, inter- val, and ratio. Nominal and ordinal scales provide less information than do interval and ratio scales, so use an interval or ratio scale whenever possible. You cannot use an interval or ratio scale in all cases because some research questions demand that a nominal or ordinal scale be used. Your choice of measurement scale should be guided by the needs of your research question. When a less informational scale must be used to preserve ecological validity, you can preserve information by creating a composite scale from a nominal and interval scale. Th is will help you to “recover” information not yielded by a nominal scale.
Beyond choosing a scale of measurement, you must also decide how to design and collect your dependent measures. Your measures must be appropriate for your subject population. Consequently, you may have to be creative when you design your mea- sures. You may count number of responses, which is a ratio scale. You can use interval scales in a variety of research applications. You must decide how to format these scales, how to present them to subjects, and how to develop clear and concise instructions for their uses.
In some research, your measure of behavior may be limited by range eff ects. Th at is, there may be an upper and lower limit imposed on your measure by the behavior of interest. For example, rats can run just so fast in a maze. Range eff ects become a problem when the behavior quickly reaches its upper or lower limit. In such cases, you may not detect a diff erence between two groups because of ceiling or fl oor eff ects. It is a good idea to conduct pilot studies to test your measures before investing the time and energy in your study. During the pilot study, you may fi nd that your measures need to be modifi ed.
Th ere are four types of dependent variables you can use in your research: behav- ioral measures, physiological measures, self-report measures, and implicit measures.
Behavioral measures include direct measures of behavior such as number of responses made or the number of errors made. Physiological measures involve measuring some biological change (e.g., heart rate, respiration rate, or brain activity). Physiological measures can be noninvasive (e.g., a PET scan) or invasive (e.g., implanting an elec- trode in a rat’s brain). Self-report measures have participants report on their own behavior and can be prospective (speculate on future behavior) or retrospective (report
bor35457_ch05_123-157.indd 156
bor35457_ch05_123-157.indd 156 4/22/13 2:52 PM4/22/13 2:52 PM
KEY TERMS 157 on past behavior). One special form of a self-report method is the Q-sort method in
which participants classify stimuli into categories. Implicit measures measure uncon- scious reactions to stimuli and are used to tap into attitudes that individuals may not admit to overtly.
Observation in psychological research diff ers from observation in the physical sciences because the psychologist deals with living organisms. Th e participants in an experiment are reactive; they may respond to more in the experimental situation than the manipulated variables. Participants bring to the experiment unique histories and attitudes that may aff ect the outcome of your experiment.
Demand characteristics can be a problem in behavioral research. Participants pick up on cues from the experimenter and research context. Th ese cues may aff ect the par- ticipant’s behavior. Furthermore, the experimenter must be careful not to inadvertently aff ect the participants. Experimenter eff ects can be avoided by using blind techniques or automating your experiment or both. Automation can be done by videotaping instructions or applying computers to control your experiment or both.
Before conducting a study, it is a good idea to do a pilot study, which is a small- scale version of your study used to test the eff ectiveness of your materials, procedures, and other parameters. You can identify and correct problems before investing time and eff ort in your main study. It is also a good idea to include manipulation checks in your research. Th ese are measures specifi cally designed to determine how your participants perceived the variables of your study. Th is information can help you to interpret your results and identify problems that may have emerged in your study.
reliability
test–retest reliability parallel-forms reliability split-half reliability accuracy
validity face validity content validity
criterion-related validity concurrent validity predictive validity construct validity nominal scale ordinal scale interval scale
ratio scale range eff ects behavioral measure physiological measure self-report measure Q-sort methodology
Implicit Association Test (IAT) demand characteristics
role attitude cues experimenter bias expectancy eff ects single-blind technique double-blind technique pilot study
manipulation check KEY TERMS
C H A P T E R
158
So far in the research process, you have made several important decisions. You have decided on a topic for your research; have taken an amorphous, broad idea; and have honed it into a tight, test- able research hypothesis. You also have made some important deci- sions about the nature of the research design that you will use, the variables you will manipulate and measure, and how you will manipu- late and measure those variables. Your next decision involves who will participate in your research study.
A number of important questions must be addressed when choosing subjects for psychological research. Should you use human participants or animal subjects? 1 How will you acquire your sample? What ethical guide- lines must you follow when using human participants or animal subjects (see Chapter 7)? If you choose human participants, what is your sample going to look like (age, race, gender, ethnicity, etc.)? If you choose to use animals, where will you get them? What are the implications of choos- ing one species or strain of a species over another? We explore these and other questions in this chapter. Th e principles discussed in this chapter apply equally to experimental and nonexperimental research. However, there are additional subject-related issues to consider if your study uses survey methodology. We discuss these issues in Chapter 9, along with other issues concerning survey research methodology.