Wiley effective methods for software testing

Introduction xxvPart I Assessing Testing Capabilities and Competencies 1Chapter 1 Assessing Capabilities, Staff Competency, and User The Three-Step Process to Becoming a World-Class Test

for Software Testing

Third Edition

William E Perry

Effective Methods for Software Testing

Third Edition

Effective Methods for Software Testing, Third Edition

Copyright © 2006 by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system or transmitted

in any form or by any means, electronic, mechanical, photocopying, recording, scanning orotherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copy-right Act, without either the prior written permission of the Publisher, or authorizationthrough payment of the appropriate per-copy fee to the Copyright Clearance Center, 222Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600 Requests to thePublisher for permission should be addressed to the Legal Department, Wiley Publishing,Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, oronline at http://www.wiley.com/go/permissions

Limit of Liability/Disclaimer of Warranty:The publisher and the author make no sentations or warranties with respect to the accuracy or completeness of the contents of thiswork and specifically disclaim all warranties, including without limitation warranties of fit-ness for a particular purpose No warranty may be created or extended by sales or promo-tional materials The advice and strategies contained herein may not be suitable for everysituation This work is sold with the understanding that the publisher is not engaged in ren-dering legal, accounting, or other professional services If professional assistance isrequired, the services of a competent professional person should be sought Neither thepublisher nor the author shall be liable for damages arising herefrom The fact that an orga-nization or Website is referred to in this work as a citation and/or a potential source of fur-ther information does not mean that the author or the publisher endorses the informationthe organization or Website may provide or recommendations it may make Further, read-ers should be aware that Internet Websites listed in this work may have changed or disap-peared between when this work was written and when it is read

repre-For general information on our other products and services or to obtain technical support,please contact our Customer Care Department within the U.S at (800) 762-2974, outside theU.S at (317) 572-3993 or fax (317) 572-4002

Library of Congress Control Number: 2005036216

Trademarks:Wiley and related trade dress are registered trademarks of Wiley Publishing,Inc., in the United States and other countries, and may not be used without written permis-sion All other trademarks are the property of their respective owners Wiley Publishing,Inc., is not associated with any product or vendor mentioned in this book

Wiley also publishes its books in a variety of electronic formats Some content that appears

in print may not be available in electronic books

taught me that testing is a lifelong process, that testing is necessary to ensure that you are meeting your objectives, and that testing can be fun

if it is performed correctly Thank you, Cynthia What you have taught me

is incorporated into many of the concepts in this book.

William E Perry holds degrees from Clarkson University, University of Rochester, andRochester Institute of Technology Bill also holds the following professional certifica-tions: CPA (Certified Public Accountant), CIA (Certified Internal Auditor), CISA (Cer-tified Information Services Auditor), CSQA (Certified Software Quality Analyst), andCSTE (Certified Software Tester) He has been an examiner for the Malcolm BaldrigeNational Quality Award, and served on standards committees for NIST (NationalInstitute of Standards and Technology), IEEE (Institute of Electrical and ElectronicsEngineers), AICPA (American Institute of Certified Public Accountants) and ISACA(Information Systems Audit and Control Association).

In 1980, Bill founded the Quality Assurance Institute (QAI), a professional tion for testers QAI offers professional certification for Quality Assurance, SoftwareTesting, Software Project Leaders and Business Analyst Professional More than 27,000individuals have been certified since the inception of the program

associa-Bill has authored more than 50 books, many published by John Wiley & Sons Herecently founded the Internal Control Institute (ICI) ICI and St Petersburg Collegerecently formed the Internal Control Center of Excellence to share best internal controlpractices, hold conferences on emerging internal control practices, and to offer e-learningcourses and a professional certification in internal control

About the Author

vi

Quality Control Technicians

John GreenoughBrian H Walls

Proofreading and Indexing

Techbooks

Introduction xxvPart I Assessing Testing Capabilities and Competencies 1Chapter 1 Assessing Capabilities, Staff Competency, and User

The Three-Step Process to Becoming a World-Class Testing

Step 1: Define a World-Class Software Testing Model 5

Customizing the World-Class Model for Your Organization 7

Step 2: Develop Baselines for Your Organization 8

Assessment 2: Assessing the Capabilities of Your Existing

Assessment 3: Assessing the Competency of Your Testers 14

Step 3: Develop an Improvement Plan 16

Part II Building a Software Testing Environment 35Chapter 2 Creating an Environment Supportive of Software Testing 37

ix

Risks Associated with Not Meeting Customer Needs 40

Writing a Policy for Software Testing 45

Testing—An Organizational Issue 50Management Support for Software Testing 50Building a Structured Approach to Software Testing 51

Chapter 3 Building the Software Testing Process 63

Guideline #1: Testing Should Reduce Software Development

Guideline #2: Testing Should Be Performed Effectively 65

Guideline #4: Testing Should Be Performed Using Business

Guideline #5: Testing Should Occur Throughout the

Guideline #6: Testing Should Test Both Function and Structure 69

Structural and Functional Tests Using Verification and

Testing That Parallels the Software Development Process 72

Customizing the Software-Testing Process 74

x Contents

Defining the Unit Test Plan Standard 83

Chapter 4 Selecting and Installing Software Testing Tools 103

Integrating Tools into the Tester’s Work Processes 103Tools Available for Testing Software 104Selecting and Using Test Tools 108

Selecting a Tool Appropriate to Its Life Cycle Phase 109

Training Testers in Tool Usage 116

Chapter 5 Building Software Tester Competency 125

What Is a Common Body of Knowledge? 125Who Is Responsible for the Software Tester’s Competency? 126How Is Personal Competency Used in Job Performance? 126

Developing a Training Curriculum 128

Chapter 6 Overview of the Software Testing Process 153

Advantages of Following a Process 153

The Seven-Step Software Testing Process 156

Using the Tester’s Workbench with the Seven-Step Process 162

Do Procedures 167

Determining the Completeness of Individual Documents 179

Task 5: Validate the Test Estimate and Project Status

Testing the Validity of the Software Cost Estimate 185Calculating the Project Status Using a Point System 189

Conducting a Walkthrough of the Customer/User Area 212

xii Contents

Task 6: Inspect the Test Plan 254

Performing Programming Phase Test Factor Analysis 326

Do Procedures 412

Applying Test Files Against Programs That Update

Creating an Inventory of Existing Project Measurements 465

Expected Versus Actual Defects Uncovered Timeline Report 472

xiv Contents

Defects Uncovered Versus Corrected Gap Timeline Report 473Average Age of Uncorrected Defects by Type Report 475

Developing Test Cases (Use Cases) Based on How

Chapter 13 Step 7: Post-Implementation Analysis 571

Do Procedures 614

Conducting the Client/Server Readiness Assessment 621Preparing a Client/Server Readiness Footprint Chart 621

Chapter 18 Testing COTS and Contracted Software 685

COTS Software Advantages, Disadvantages, and Risks 686

Step 2: Integrate the Software into Existing Work Flows 698

xviii Contents

Task 4: Acceptance-Test the Software Process 702

Modifying the Testing Process for Contracted Software 704

Task 4: List Structural Components Affected by the Platform(s) 723

Characteristics of Security Penetration 756

Evaluating the Adequacy of Security 761

Task 2: Select Web-Based Tests 807

Building an Agile Testing Process 820

Chapter 24 Building Agility into the Testing Process 831

Step 1: Measure Software Process Variability 831

Improvement Shopping List 856

Step 3: Build on Strength, Minimize Weakness 857

Step 4: Identify and Address Improvement Barriers 861

Step 5: Identify and Address Cultural and Communication

Determining What Can Be Done in the Current Culture 879Determining the Desired Culture for Time Compression 879

Step 7: Develop and Execute an Implementation Plan 891

Most books about software testing explain “what” to do This book, on the other hand,takes more of a “how-to” approach It provides the procedures, templates, checklists,and assessment questionnaires necessary to conduct effective and efficient softwaretesting.

The book is divided into five parts, as follows:

■■ Part One: Assessing Testing Capabilities and Competencies.It is difficult to

make any significant change until you know where you are A baseline tells notonly where you are, but lets you measure your progress as your testing strate-

gies and techniques improve Part One provides three baseline assessments: thecapabilities of your software testing group, the competencies of your individ-

ual testers, and the effectiveness of your test processes

■■ Part Two: Building a Software Testing Environment.Software testers are

most effective when they work in an environment that encourages and supports

well-established testing policies and procedures The environment includes the

procedures and tools for testing, as well as the support and encouragement of

management Part Two begins by describing how to build an environment

con-ducive to testing, and then expands the discussion by describing how to develop

a testing process, select testing tools, and build the competency of your testers

■■ Part Three: The Seven-Step Testing Process.Part Three comprises the core

material in the book It defines a world-class software testing process, from its

initiation through testing changes made to operational software systems This

material can be used two ways First, it contains sufficient procedures and plates so that an organization can use the process as their own Of course, mostorganizations inevitably will make some changes to accommodate local vocab-

tem-ulary, specific needs, and customs This customization process, the seven-step

process in this book becomes “owned” by the software testers

xxv

■■ Part Four: Incorporating Specialized Testing Responsibilities.The seven-steptesting process is a generic process that almost all software testing organiza-tions can use However, the mission of software testers may incorporate spe-cialized activities, such as testing security Rather than incorporating thesespecialized testing activities directly into the seven-step process, they are pre-sented as individual, specialized activities As appropriate, they can be incor-porated into the seven-step process.

■■ Part Five: Building Agility into the Testing Process.Part Five, which draws

on what you’ve learned earlier in the book, is designed to help you identify thestrengths and weaknesses of your current software testing process, and thenmodify it to become more usable or agile

Getting the Most Out of This Book

This book is not designed to be read like a novel, from beginning to end, nor is it filledwith human interest stories about testers The book focuses on how to conduct softwaretesting It is designed to help you improve your testing competencies and processes Theself-assessments in Part One will help you identify which parts of the book you need toread first

The following guidelines will help you maximize the benefit from this book:

■■ Establish a baseline of current performance.Part One of this book (and ter 5) contains four self-assessments for establishing baselines You need toknow where you are so that you can develop a good plan for moving forward

Chap-■■ Define the software testing organization you would like to have.It has beensaid that if you do not know where you’re going, all roads lead there Too manysoftware testing groups just add new testing programs, processes, and toolswithout knowing if they will integrate effectively

■■ Develop a plan for moving from your baseline to your goal.Few tions can quickly and effectively install an entirely new software testing

organiza-process Gradual change is normally much better than radical change fore, identify the gaps between where you are and where you want to be.Determine which of those gaps if closed would provide the greatest benefit toyour organization That becomes the part of the plan you implement first Overtime you will move the entire testing process from your current baseline toyour desired goal

There-For additional information on software testing conferences and training programs,visit www.taiworldwide.org For information on software testing certifications,visit www.softwarecertifications.org

What’s New in the Third Edition

The core of this book is the step-by-step process for testing software This edition hassimplified that process from 11 steps to 7 steps

xxvi Introduction

A major addition to this edition is the self-assessment in Chapter 5, which testers canuse to identify their strengths and weaknesses and then build a personal improvementplan The self-assessment is based on the Common Body of Knowledge (CBOK) for theCertified Software Tester (CSTE).

Other significant additions include

■■ A new chapter on testing internal control

■■ An expanded chapter on testing security

■■ A new chapter on adapting testing to the developmental methodology used to

build the software

■■ Two new chapters on how to incorporate agile methods into the testing process

What’s on the CD

This book includes a CD that contains the work papers and quality control checklists

to help you implement the software testing process

To use the CD, first you need to select a software testing activity that you want toimplement in your organization—for example, test planning Then, from the chapter

on test planning, identify those work papers and checklists that you believe would bebeneficial to your organization You can extract those work papers and checklists fromthe CD and begin a customization process For example, you can include the name ofyour organization, add or delete portions of the work papers, and change the termi-nology to be consistent with your organization

After you have used the work papers for conducting a software test, you shouldbundle the work papers into a case study for new testers If they use the book to learnthe basics of software testing and then can cross reference what they have learned toexamples of how the work papers are actually used in software testing, learningshould be accelerated

for Software Testing

Third Edition

Assessing Testing Capabilities and Competencies

It has been said, “If you do not know where you are going, all roads lead there.” ditionally, many IT organizations annually develop a list of improvements to incorpo-rate into their operations without establishing a goal Using this approach, the ITorganization can declare “victory” any time it wants

Tra-This chapter will help you understand the importance of following a well-definedprocess for becoming a world-class software testing organization This chapter willhelp you define your strengths and deficiencies, your staff competencies and deficien-cies, and areas of user dissatisfaction

The objective of this chapter is threefold: to define a world-class software testingmodel, to provide a self-assessment process for your software testing organization tomeasure yourself against the world-class model, and to provide some planning con-siderations for moving to a world-class level

The Three-Step Process to Becoming a

World-Class Testing Organization

The roadmap to become a world-class software testing organization is a simple step process, as follows:

three-1 Define or adopt a world-class software testing model

2 Determine your organization’s current level of software testing capabilities,

competencies, and user satisfaction

Assessing Capabilities, Staff

Competency, and User

Satisfaction

1

3 Develop and implement a plan to upgrade from your current capabilities, petencies, and user satisfaction to those in the world-class software testingmodel.

com-This three-step process requires you to compare your current capabilities, tencies, and user satisfaction against those of the world-class software testing model.This assessment will enable you to develop a baseline of your organization’s perfor-mance The plan that you develop will, over time, move that baseline from its currentlevel of performance to a world-class level Understanding the model for a world-classsoftware testing organization and then comparing your organization will provide youwith a plan for using the remainder of the material in this book

compe-Software testing is an integral part of the software-development process, whichcomprises the following four components (see Figure 1-1):

1 Plan (P): Devise a plan Define your objective and determine the strategy and

supporting methods to achieve it You should base the plan on an assessment

of your current situation, and the strategy should clearly focus on the strategicinitiatives/key units that will drive your improvement plan

2 Do (D): Execute the plan Create the conditions and perform the necessary

training to execute the plan Make sure everyone thoroughly understands theobjectives and the plan Teach workers the procedures and skills they need tofulfill the plan and thoroughly understand the job Then perform the workaccording to these procedures

3 Check (C): Check the results Check to determine whether work is progressing

according to the plan and whether the expected results are being obtained.Check for performance of the set procedures, changes in conditions, or abnor-malities that may appear As often as possible, compare the results of the workwith the objectives

4 Act (A): Take the necessary action If your checkup reveals that the work is not

being performed according to the plan or that results are not what you pated, devise measures to take appropriate actions

antici-Figure 1-1 The four components of the software-development process.

PLAN

CHECK

DO ACT

Testing involves only the “check” component of the plan-do-check-act (PDCA)cycle The software development team is responsible for the three remaining compo-nents The development team plans the project and builds the software (the “do” com-ponent); the testers check to determine that the software meets the needs of thecustomers and users If it does not, the testers report defects to the development team.

It is the development team that makes the determination as to whether the uncovereddefects are to be corrected

The role of testing is to fulfill the check responsibilities assigned to the testers; it isnot to determine whether software can be placed into production That is the responsi-bility of the customers, users, and development team

Step 1: Define a World-Class

Software Testing Model

There is no generally accepted model for a world-class software testing organization.However, analyzing the best testing organizations among the more than 1,000 IT orga-nizations affiliated with the Quality Assurance Institute (QAI) enabled QAI to identifythe attributes of the best software testing organizations (see Figure 1-2) Organizationsthat follow this model report more effective and efficient testing than those that do not

Figure 1-2 Model of a world-class software testing organization.

Stakeholders

Satisfaction

TestProcesses

TestProcessImprovement

Management

of TestingTesting Strategic

Dashboard

Testing TacticalDashboard

The world-class software testing model includes

■■ Test environment.The conditions that management has put into place thatboth enable and constrain how testing is performed The test environmentincludes management support, resources, work processes, tools, motivation,and so forth

■■ Process to test a single software project.The standards and procedures testersuse to test

■■ Tester competency.The skill sets needed to test software in a test environment.The three self-assessments that follow are for the above three attributes of aworld-class software testing organization

three attributes of a world-class software testing organization

The world-class model of a software testing organization focuses on stakeholder isfaction This assumes a greater role for a world-class software testing organizationthan just testing against documented software requirements Chapter 2 defines themany roles that software testing can adopt; however, those roles include much morethan testing documented software requirements They include testing for quality fac-tors such as ease of use, meeting testing schedules and budgets, and minimizing therisks involved with any software project

sat-According to the world-class model, the following parties have a vested interest insoftware testing:

■■ Software customer.The party or department that contracts for the software to

be developed

■■ Software user.The individual or group that will use the software once it isplaced into production (Note: This may be the customer or it may be partiesother than the customer.)

■■ Software developer.The individual or group that receives the requirementsfrom the software user or assists in writing them, designing, building, andmaintaining the software, as needed

■■ Development tester.The individual or group that performs the test functionwithin the software development group

■■ IT management.The individual or group with responsibility for fulfilling theinformation technology mission Testing supports fulfilling that mission

■■ Senior management.The CEO of the organization and other senior executiveswho are responsible for fulfilling the organization mission Information tech-nology is an activity that supports fulfilling that mission

■■ Auditor.The individual or group responsible for evaluating the effectiveness,efficiency, and adequacy of controls in the information technology area Testing

is considered a control by the audit function

6 Chapter 1

■■ Project manager.The individual responsible for managing the building,

main-taining, and/or implementing of software

The test mission, strategy, and environment must be focused on stakeholder faction The mission defines the testing objectives; the strategy defines how the missionwill be accomplished; and the environment provides the culture, processes, and toolsthat are conducive to effective and efficient software testing

satis-The test processes are those step-by-step procedures that the testers will follow toaccomplish their assigned tasks Test processes executed by trained and competenttesters enable those testers to accomplish the defined test mission

The test processes need to be improved continually for two reasons: to make themmore effective and efficient to use, and to incorporate updated approaches into testingnew technologies and software development methodologies

The responsibility for ensuring that the execution of the test processes meets thedefined test mission lies with management Management must ensure that testers arefollowing and can accomplish the test plan, and that the plan will, in fact, accomplishthe test objectives If not, management should modify the plan to meet those objectives.Management and testers need tools to enable them to fulfill their responsibilities.Two very important tools are the testing strategic dashboard and the testing tactical

dashboard The testing strategic dashboard includes key indicators such as user tion, staff competency, and the percent of tests completed The testing tactical dashboard

satisfac-includes test indicators such as the number of requirements tested and percent correct,defects uncovered, defects corrected and uncorrected, and the schedule and budgetstatus

Management must ensure that if you meet the testing tactical key indicators, youwill, in fact, meet the objectives defined by the strategic key indicators

Customizing the World-Class Model for Your

Organization

You can customize the world-class model for software testing by defining the utes of each of its components (refer to Figure 1-2) The material in this book explainsthe attributes of all the components: stakeholder satisfaction, test mission, test man-agement and enabling competencies are discussed in Part 2 The test processes areexplained in Parts 3 and 4 Test process improvement is described in Part 5 of this book

attrib-As you read those parts of the book, you can customize those attributes based on themission of your organization For example, in describing a tester’s competency, skill setsfor testing COTS software and outsourced software will be listed However, if your orga-nization does not use COTS software or does not outsource the development of software,you would not need those skills in your testing staff Likewise, if your testers are notresponsible for testing security, you would not need a test processes for testing security The three self-assessments included in this chapter are based on the model in Figure1-2 However, it is recognized that few testing organizations need all these testingcapabilities and competencies Therefore, you need to develop the model that is suited

to your test mission

Step 2: Develop Baselines for Your Organization

This section presents the following three self-assessment categories to enable you tocompare your testing organization against the world-class model:

1 Assessing the test environment This includes user satisfaction, management

support, environment, planning, tools, test processes, measurement, qualitycontrol, and training

2 Assessing the process for testing individual software projects This category

of assessment will assess your testing process against the seven-step process fortesting individual software projects presented in Part 3 of this book

3 Assessing the competencies of software testers This self-assessment will be

based on the 2006 Common Body of Knowledge (CBOK) developed by the Certification Board of the Software Certifications Organization Each of the recommended ten competencies for software tester will be assessed A moredetailed assessment to be used in individuals to compare their specific testcompetencies against the 2006 CBOK is provided in Chapter 5

Assessment 1: Assessing the Test Environment

During the past 25 years, the Quality Assurance Institute (QAI) has been studyingwhat makes software testing organizations successful As a result, QAI has identifiedthe following eight criteria:

Test environment planning

diagram indicating the areas to address, called drivers, which results in a world-class

testing organization

8 Chapter 1

Figure 1-3 Overview of the testing environment.

Software testing organizations can use the results of this assessment in any one ofthe following three ways:

1 To determine their current testing environmental status versus the environment

of a world-class testing organization The responses to the items address will

indicate an organization’s strengths and weaknesses compared to the

environ-ment of a world-class testing organization

2 To develop the goal/objectives to accomplish becoming a world-class testing

organization QAI’s world-class criteria indicate a profile of the environment of

a world-class testing organization Achieving those objectives can lead you to

become a more effective software testing organization

3 To develop an improvement plan

By doing the assessment, you will develop a Footprint Chart that shows whereimprovement is needed Those criteria in which you are deficient become the meansfor improving the environment of your software testing organization

Implementation Procedures

This practice involves the following four tasks:

■■ Build the assessment team

■■ Complete the assessment questionnaires

■■ Build the footprint chart

■■ Assess the results

Test Tools

Use of Test

Processes

M ana ge m

ent Support

Test E nvi

ro nm

ent

Pl ann

ing

Test T ra

Use

r Satisfactio n

Test Me asu re

ent

ntrol

World-Class Testing