EPI Case Study 2 Reliability, Validity, and Tests of Agreement Student Version 1

Tuberculosis Screening Time to Complete Exercise: 30 minutes LEARNING OBJECTIVES At the completion of this Case Study, participants should be able to:  Know how to construct 2-by-2 tab

EPI Case Study 2: Reliability, Validity, and Tests of Agreement in

M Tuberculosis Screening

Time to Complete Exercise: 30 minutes

LEARNING OBJECTIVES

At the completion of this Case Study, participants should be able to:

 Know how to construct 2-by-2 tables

 Compare screening techniques using the Kappa Statistic

 Distinguish between reliability and validity

 Identify groups at high risk for TB infection

 Identify methods used to screen for TB infection

ASPH EPIDEMIOLOGY COMPETENCIES ADDRESSED

C.2 Identify the principles and limitations of public health screening programs

C 3 Describe a public health problem in terms of magnitude, person, place, and time

C 6 Apply the basic terminology and definitions of epidemiology

C 7 Calculate basic epidemiologic measures

C 9 Draw appropriate inference from epidemiologic data

C 10 Evaluate the strengths and limitations of epidemiologic reports

Suggested citation: New Jersey Medical School Global Tuberculosis Institute

/Incorporating Tuberculosis into Public Health Core Curriculum./ 2009: Epidemiology

Case Study 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis

Screening STUDENT Version 1.0.

This material was developed by the staff at the Global Tuberculosis Institute

(GTBI), one of four Regional Training and Medical Consultation Centers funded

by the Centers for Disease Control and Prevention It is published for learning

purposes only Permission to reprint excerpts from other sources was granted

Case study author(s) name and position:

George Khalil, MPH (work done as MPH candidate)

Marian R Passannante, PhD

Associate Professor, University of Medicine & Dentistry of New Jersey, New

Jersey Medical School and School of Public Health

Epidemiologist, NJMS, GTBI

For further information please contact:

New Jersey Medical School Global Tuberculosis Institute (GTBI)

225 Warren Street P.O Box 1709 Newark, NJ 07101-1709

or by phone at 973-972-0979

In the United States, no vaccines are given to prevent the transmission of tuberculosis (TB) due to their current lack of efficacy.1 Because scientists are still working to create a more efficient vaccine, the generally accepted approach to TB control relies on screening,

surveillance, and contact investigations.2

Identifying and treating persons with latent TB infection (LTBI) at high risk for

developing TB is part of the current TB elimination strategy in the United States Screening is essential in the identification phase of this strategy (2) The most common method of

screening is with purified protein derivative (PPD), a type of tuberculin skin test (TST)

Infection with Mycobacterium tuberculosis (M tuberculosis) produces a delayed-type

hypersensitivity reaction to certain antigenic components of the organism that are contained

in extracts of culture filtrates called tuberculins The skin test is injected in the forearm and read by a trained clinician after 48 to 72 hours The size of the reaction is measured in

millimeters and interpreted according to its size, using cutoff points corresponding to the degree of induration.3 Another type of screening test is called interferon gamma assay (IGA), which measures the production of the cellular interferon gamma by T-cells after sensitization

with M tuberculosis antigens Although researchers believe that interferon tests are

preferable to the TSTs, they are much more expensive.4

This exercise will be based on the following study Gerald H Mazurek; Philip A

LoBue; Charles L Daley; John Bernardo; Alfred A Lardizabal; William R Bishai;

Michael F Iademarco; James S Rothel, Comparison of a Whole-Blood Interferon

Assay With Tuberculin Skin Testing for Detecting Latent Mycobacterium tuberculosis

Infection

JAMA 2001;286:1740-1747.5 Sections of this document have been reprinted with

permission of the journal (permission pending).

Context Identifying persons with latent tuberculosis infection (LTBI) is crucial to the goal of TB elimination A whole-blood interferon (IFN- ) assay, the

Quanti-FERON-TB test, is a promising in vitro diagnostic test for LQuanti-FERON-TBI that has potential advantages over the tuberculin skin test (TST)

Objectives To compare the IFN- assay with the TST and to identify factors

associated with discordance between the tests

Design and Setting Prospective comparison study conducted at 5 university affiliated

sites in the United States between March 1, 1998 and June 30, 1999

Participants A total of 1226 adults (mean age, 39 years) with varying risks of

Mycobacterium tuberculosis infection or documented or suspected active TB, all of

whom underwent both the IFN- assay and the TST.

Main Outcome Measure Level of agreement between the IFN- assay and the

TST

The study was conducted at 5 sites: Boston University School of Medicine, Mass; Johns Hopkins School of Hygiene and Public Health, Baltimore, Md; University of California at San Francisco; New Jersey Medical School, Newark; and University of California at San Diego, using a common protocol These sites were randomly coded

as A-E in the analysis Ethical approval for the study was obtained from the

institutional review boards at the Centers for Disease Control and Prevention (CDC), which supported the study, and the 5 study sites prior to enrolling any subjects All participants provided written informed consent

Persons recruited for the study were 18 years or older and included persons

requesting a preemployment or preschool enrollment TST; persons being screened with a TST because they were considered to be at high risk for LTBI; persons in whom

TB was clinically suspected and who had received fewer than 6 weeks of anti-TB therapy; and persons who previously had active TB, confirmed by a positive culture, and who had completed a course of multidrug anti-TB therapy within the prior 2 years Subjects were excluded from the study if they self-reported as pregnant or

HIV-positive; had a history of severe reaction to tuberculin; were immunocompromised due

to leukemia, lymphoma, or Hodgkin disease; or had taken immunosuppressive drugs (eg,orticosteroids,methotrexate, azathioprine) during the preceding 3 months

After providing written informed consent, enrolled persons completed a detailed

questionnaire about possible risk factors for exposure to M tuberculosis Subjects were also asked to indicate results of any prior TST, whether they had received BCG

vaccination, details of any contact with a person having TB, any risk factors associated with HIV infection, and whether they had any other medical conditions When

applicable, data were also collected from medical records about findings on chest radiography, results and dates of cultures for mycobacteria, and details of treatment for TB Data were collected on subjects’ age, race, place of birth, residence outside of the United States, and residence or work (paid or unpaid) in a health care setting, prison, homeless shelter, drug rehabilitation unit, or other group housing Based on responses to the questionnaire and a review of available medical records, persons were categorized into 4 study groups: (1) low-risk for LTBI, subjects receiving

preemployment or preschool enrollment TST with no identified risks for LTBI; (2) high-risk for LTBI, asymptomatic subjects with high-risk of LTBI including contacts of patients with TB; persons from countries where tuberculosis is prevalent (>10 cases per

100000 population)26; intravenous drug users; persons who lived, worked, or

volunteered on a regular basis in a homeless shelter, prison, drug rehabilitation unit, hospital, or nursing home; and persons determined to be at increased risk by prior local investigations;(3) TB suspects, subjects being evaluated for active TB who had received fewer than 6 weeks of anti-TB therapy; and (4) culture-confirmed TB,

subjects who completed treatment for culture-confirmed TB within the prior 2 years

However, to maintain the integrity of group 1 as truly low risk for LTBI, persons

considered to be at high-risk for LTBI at enrollment were assigned to group 2 even when risk factors were denied

Reference 26 in Mazurek et al: World Health Organization Global Tuberculosis

Control Geneva, Switzerland: WHO; 1999.WHO/ CDS/CPC/TB/99.259.

For the purposes of this exercise, we will examine the data from Group 1 (low-risk group for

TB infection) and Group 2 (high-risk group) Table 1 compares the responses to both TST and IFN tests for Groups 1 (low risk) and 2 (high risk)

Table 1 Response to TST and IFN- tests in high- and low-risk Groups5

Question 1

A. For Group 1, create a 2X2 table to test the association between TST and IFN-

readings Show percent for marginal totals

B. For Group 2, create a 2X2 table to test the association between TST and IFN- readings Show percent for marginal totals

Use the following 2x2 tables:

A

Positive Negative

Total Positive

Negative

Total

B

Positive Negative

Total Positive

Negative

Total

Question 2

A. For Group 1, calculate an overall percent agreement by TST and IFN- assay and interpret

B Do the same for Group 2

Note: Percent agreement can be calculated as (a+d)/(a+b+c+d) x 100 and is called po (or proportion of agreement observed) where a,b,c,d are shown in this table:

Outcome Independent

variable

Yes No Total

A po or % agreement for Group 1 =

B po or % agreement for Group 2 =

Question 3

For Groups 1 and 2, create2X2 tables of percent agreement by screening test expected by chance alone and calculate its percent agreement To do this you can calculate the

expected values for cells a and d and then calculate the percent agreement expected by chance using the formula (a expected value + d expected value)/(a+b+c+d) x 100 This is called pe (or proportion of agreement expected)

Remember: Expected values= (row total x column total)/grand total

Use the following 2x2 tables:

A Percent agreement expected by chance alone for Group 1=

Positive Negative

Total Positive

Negative

Total

B Percent agreement expected by chance alone for Group 2=

Positive Negative Positive

Negative

Total

Question 4

A What is the kappa statistic of the 2 tests for Group 1 (low risk for TB)?

B And for Group 2 (high risk for TB)?

κ = (percent observed agreement) – (percent agreement by chance alone)

100%- (percent agreement expected by chance alone)

Table 26

Source: Understanding interobserver agreement: the kappa statistic Viera AJ, Garrett JM Fam Med 2005;37:362 Permission granted to reprint table

Question 5

Rate the overall reliability of the screening tests Does prevalence seem to have an impact on the kappa value?

Question 6

Is there a limitation in comparing the IFN- assay with the TST? If so, what is it?

Validity and Reliability

A useful screening test is both reliable (reproducible or accurate) and valid (precise) In Figure 16, assuming the goal is to hit the middle of the target, B and D are reliable (also called reproducible) but only D is both reliable and accurate Unfortunately, to test for validity the truth has to be known or there has to be a “gold standard” and in TB screening there is no gold standard So, in this exercise we are only looking at reliability of two screening tests

Source: Understanding interobserver agreement: the kappa statistic Viera AJ, Garrett JM Fam Med 2005;37:361 Permission granted to reprint Figure 1

Works Cited

1 Development of new vaccines for tuberculosis MMWR Recomm Rep 1998;47

(RR-13):1-6

2 The role of BCG vaccine in the prevention and control of tuberculosis in the United States:

A joint statement by the Advisory Council for the Elimination of TB and the Advisory

Committee on Immunization Practices MMWR Recomm Rep 1996;45 (RR-4):1-18

3 Ayub A, Yale SH, Reed KD, Nasser RM, Gilbert SR Outpatient practice management tips

Testing for Latent Tuberculosis Accessed February 8, 2008

http:www.clinmedres.org/cgi/content/full/2/3/191

4 Madariaga MG, Jalali Z, Swindells S Clinical utility of interferon gamma assay in the

diagnosis of tuberculosis J Am Board Fam Med 2007;20:540-547.

5 Mazurek GH, LoBue PA, Daley CL, et al Comparison of a whole-blood interferon gamma assay with tuberculin skin testing for detecting latent mycobacterium tuberculosis infection

JAMA 2001;286:1740-1747

6 Viera J, Garrett JM Understanding interobserver agreement: the kappa statistic Fam Med.

2005;37:360-363