Fagan inspection method

Giới thiệu phương pháp kiểm tra Fagan

Fagan Inspection Method

1

Workshop on Software

Inspection

2

  Myths of Peer Reviews

  Need for Software Inspection

  Exercise: Preparation Time

  Exercise: Defect Logging Meeting

  Exercise: Causal Analysis Meeting

  Exercise: Summary Presentation

  Inspection metrics

  How to make Inspection happen

  Closing Session

4

facets of Quality Management and Process Improvement. 

  Practitioner of Fagan Inspection for more than 15 years and

conducted Inspection training to many prestigious clients in India,

Japan and Vietnam

  Process improvement consultant, who helps clients in the

countries like India, USA, Japan, China, Taiwan, Brazil, Australia,

Malaysia, Vietnam, etc

  Visiting faculty in many universities and business schools including

Great Lakes Institute of Management, Chennai, India

  Lean Six Sigma Corporate Champion, Certified Scrum Master,

Certified CMMI Lead Appraiser, and a passionate trainer. 

  Earlier jobs include Vice President of Corporate Quality at HCL

Technologies and Manager, Quality at Oracle India

  Holds ME (Industrial Engineering) from PSG College of

Technology and Exec MBA from Great Lakes Institute of

Management

  Raja can be reached on his email: mrajamanickam@gmail.com

4

5

 Participate and share

 Don’t reserve your questions at any time

 Maintain punctuality during the start and

breaks

 No web browsing/mails/skype during the

the session pls!

 Mobiles in silent mode?

 Anything else to be added?

 Let us understand your specific

expectations from this workshop

8

  Verify that specifications are satisfied

  Verify conformance to standards

  Identify deviation from standards and

specifications

  Collect data for improvement

  Reviews waste lot of time

  Reviews are costly

  Hard work alone results in quality

  Everyone always tries to produce high

quality products

  Reviews can be done only by Senior

Staff

13

14

Source : Software Inspection - An Industry Best Practice, IEEE Computer Society Press, 1996

Catching Defects early in the Development Process is

Economical

16

Requirements

Coding Unit Test

Integration Test System Test

Design

  Testing starts too late in the development process

  In testing, you find only symptoms, not defects

  Testing is expensive

How about Testing?

17

  According to studies, Inspection is three times more effective

in finding defect than testing

  Inspection finds most of the problems testing would find, and does so more

efficiently

  However, some subtle bugs may escape inspection, and could

only be detected through testing by executing the software

Without Inspections

240 Total Defects

10 Defects goes to the customer

Source : Software Inspection - An Industry Best Practice, IEEE Computer Society Press, 1996

Requirements 20

Design 40

Unit Test 50 Integration Test 20 System Test 10

Reduced Defects (Total 41)

Improved Product Quality

Reduced Rework Costs

Source : IBM Corp., Software Product Assurance, NY, 1987

Improved Product Quality

Reduced Rework Costs

Source: A history of Software Inspections – Michal Fagan

22

  Introduction of Inspections in each phase of

the Development Process reduces Defects

and Rework Costs

  Inspection Process works regardless of

o  Type of Software

o  Type of Design

  Calendar time reduction (10% to 30%)

  Reduction in development effort (9% to 40%)

  Reduction of testing costs and time

  Maintenance cost reduction (35% to 65%)

Raytheon

  Reduced "rework" from 41% to 20%

  Reduced effort to fix integration problems by 80%

IBM

  1 hour of inspection saved 20 hours of testing

  Saved 82 hours of rework in released product

  Inspections Resulted in

  23% increase in coding productivity

  38% reduction in defects detected after unit test

AT & T

  Inspections Resulted in

25

IBM Santa Teresa Lab

  3.5 hours to find bug with inspection, 15-25 through

◦  80 % of defects detected by Inspections were unlikely to be

detected by other methods

  Review the product, not the producer

  Use standards to avoid disagreement over style

  Set an agenda and maintain it

  Identify problems, but don’t attempt to solve

them

  Take written notes

  Limit the number of participants

28

  Develop a checklist for each product to be

reviewed

  Review complete pieces of a product

  Insist upon advance preparation

  Allocate resources and time schedule

  Avoid consecutive Peer Reviews

  Review your early reviews

29

 Offline Review

 Peer Review – Walkthrough

 Peer Review – Inspection

Work

Product Review Type

  Depends on reviewers domain knowledge

and ability to spot defects

  Considered as NOT part of reviewers regular

activity

  Not very effective process

  Not repeatable

Author explains fixes to Review team

& his Manager

Author calls for another walkthrough meeting & the cycle continues

  “None of us are as good as all of us”

  More effective than offline review

37

  No advance preparation by the reviewers

 Review team understands and reviews the work

product simultaneously

  Team may lead to diverse controversial issues

  Get to know ONLY the author’s view of the

work product

  Author leads the show

38

Inspection Medium - High >60%

Walkthrough Medium <50%

Offline Review Low <35%

Self Checking Low <20%

Reasons:

More formality and rigor

Well defined Inspection Methodology

All participants are active and responsible

Requires preparation

Repeatable process

Source: Capers Jones, Software Measurement and Estimation

  Reviews reduce rework

  Offline is the simplest one

 To be used for less critical work products

  Walkthrough

 More Effective compared to Offline reviews

 Often used for code reviews

  Inspections

 The most beneficial (to be discussed extensively now on)

43

A formal process requiring advance

preparation, planned and scheduled as mile

stone to detect defect in work products

Rework Follow-Up Defect Log

Meets Exit Criteria for the work?

Yes

No

Defect Logging Meeting Causal Analysis Meeting Improvement Process

  Emphasis on advance preparation

  Inspection Process designed to provide

continuous process Improvement

46

Guidelines on :

  Duration on Inspection Meeting

  Size of Inspection team

  Size of documents that can be inspected in one

meeting

47

  Inspect the Work Product - Not the Producer

  Focus on detecting as many defects as possible

  Choose the right participants

  No management participation (even if the

manager participates, he acts only as a ‘peer’)

  Focus on playing the assigned roles

  Promptly follow the start & end time for all the

meetings

  Spend quality time on preparation

  Leave the egos at the door of the Inspection meeting

  Limit the inspection meeting time (not to exceed 2

hours)

  Inspector’s Job is only to report defects (do not try to

solve them)

  Author - & only Author - is responsible for fixing defects

  Inspection meeting discussion is only to make author

understand the issue

  Discussion on Inspection to be confined to Inspection

o  Keeps the team Focused

o  Not necessarily the senior most member of

the Inspection Team

o  Plays an active role as an Inspector

52

  Author

o  Developer of the work product being Inspected

o  Plays an active role as an Inspector

  Reader

o  Paraphrase the work product in the defect

logging meeting to provide an exact view of the

work product

53

o  Plays an active role as an Inspector

o  Usually belongs to the same development team

o  Classify errors (must know classification)

o  After each recording, read out to inspection

team for confirmation

o  Read out/present full list to inspection team

  Process creates at least one person who is as

well informed about the work product as the

author

  Planning

  Overview Meeting

  Preparation

  Defect Logging Meeting

  Causal Analysis Meeting

  Rework

  Verification

58

  Choose team, materials, dates

  Present product, process, goals

  Check work product, note defects

  Consolidate and log defects

  Analyze causes of systemic defects

o  Moderator, Author

o  Plan the various steps in the Inspection Process

for the work product being Inspected

o  Typical Time

30 Minutes

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

  Objectives

◦ Gather review package: work product, checklists,

references, and data sheets

◦  Form inspection team

◦  Determine dates for meetings

  Procedure

◦  Author identifies Moderator

◦  Author & Moderator identify the team

◦  Moderator checks work product for readiness

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

61

  Overview Meeting

o  Moderator, Author, Reader, Inspector

  Ensure that all the participants understand their

roles and roles of others in the Inspection team

  Ensure that all the participants understand the work

product being Inspected

  Typical Rate

500 LOC/Hr for code

600 Lines/Hr for Text

Planning OV meeting Preparation CA Mtg Verify

62

  Objectives

◦  Author provides overview

◦  Reviewers obtain review package

◦  Preparation goals established

◦  Reviewers commit to participate

  Procedure

◦  Author presents overview, if necessary

◦  Moderator distributes review package

◦  Reader, Recorder role for DL Meeting assigned

◦  Moderator reviews preparation procedure

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

 Preparation

o  Moderator, Author, Reader, Inspector

  Learn the work product

  Inspect the work Product

  Document defects found in the work product

64

 Objectives

◦  Find maximum number of non-minor issues

 Procedure

◦  Allocate recommended time to preparation

◦  Perform individual review of work product

◦  Use checklists and references to focus attention

◦  Note Major, Minor and Trivial issues on Inspector Form

◦  Note issues and clarify items

Planning OV meeting Preparation DL Mtg CA Mtg Verify

65

  Major

◦  Defects that may cause the system to hang, crash,

produce incorrect results or behavior, or corrupt

user data No known work-arounds

  Minor

◦  Defects that cause incorrect results or behavior

with known work-arounds Large and/or important

areas of the system is affected

  Trivial

◦  Defects that affect limited areas of functionality that

can either be worked around or ignored

 Why not write on the work product?

Advantages of Inspector Form:

◦  Inspectors articulate issues clearly during

preparation, saving meeting time

◦  Preparation statistics gathering simplified

◦  Preparation effectiveness (% true defects, %

redundancy) and checklist effectiveness is

measurable

◦  Defects are pre categorized saving the DL

67

 Disadvantages of Reviewer Data Sheet:

◦  Requires extra time (15 minutes?)

◦  Discourages last minute preparation

◦  Makes quality of preparation more visible

68

  Defect Logging Meeting

o  Moderator, Author, Reader, Inspector

  Document Unique defects found by the participants

◦  Provide opportunity for group synergy

◦  Improve reviewing skill by observing others

◦  Create shared knowledge of work product

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

70

 Procedure

◦  Reader paraphrases the work product

◦  Reviewers raise issues

◦  Recorder notes defects on Defect Logging Sheet

◦  Moderator decides on ‘resolution’

◦  Moderator forms CAT

71

 Objectives

◦  Arrive at a phase of origin and causes of the defects and

to suggest process improvements

 Procedure

◦  Inspection Team arrive at the phase of origin of all defects

◦  Inspection Team categorize the defect into ‘systemic’ and

‘mis execution’ and analyze causes of ‘systemic defects’

◦  Recorder records the info in Defect Logging sheet

◦  Recorder sends the process improvement info to Quality

◦  Assess each defect and fix

◦  Update Defect logging sheet

◦  Note time taken for rework

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

73

 Procedure

◦  Author obtains Defect Logging Sheet

containing consolidated defects

◦  Author assesses each defect and analyses

action taken

◦  When finished Author provides Defect

Logging sheet and reworked product to CAT

to Verify

74

 CAT

 Objectives

◦  Assess the (reworked) work product quality

◦  Assess the inspection process

◦  Pass or fail the work product

Planning OV meeting Preparation DL Mtg CA Mtg Rework Verify

 Procedure for CAT:

◦  Obtain reworked product and DL Sheet

◦  Review work product/DL sheet for problems

◦  Provide recommendation for work product

◦  Perform sign-off

◦  Moderator compute summary statistics for

inspection

◦  Moderator sends DL sheet to quality Team

◦  Moderator informs the Inspection team of

closure

76

 All advantages of Walkthroughs Plus

  Stress on advance preparation (Heart of Inspection)

  Well defined responsibilities for all roles

  Three different perspectives of the same work product

  Moderator runs all meetings

  High return on invested time and effort

  Immediate feedback to developers

  Repeatable Process

  Avoids injecting defects in future work

  Part of training for new people and back up for

  Mistakes as too ‘low tech’ to be effective

  Dependencies of members’ presence in

meetings

  Resistance from troops on implementation of

this method

All these can be resolved through good formal

training on this process and selected implementation

to get buy in

- at various phases

79

 Moderator

◦  Ensure that the work product is ready for Inspection

◦  Select Inspection team with the help of author

◦  Get Commitment from team members to:

  Play respective roles

  Invest required amount of time & effort

◦  Schedule Overview & defect logging meetings

80

◦  Distribute package for inspection containing :

  Work Product to be Inspected

  Input documents

  Support Documents

  Standards

  Checklist

  Inspection Meeting notice form

◦  Ensure that the material to be used in the overview

meeting is sufficient

◦  Keep a log of total time spend on planning

 Author

◦  Ensure that the work product is ready for Inspection

◦  Help the moderator in selecting the Inspection team

◦  Prepare a presentation for the overview meeting

  Explain context for functioning the product

  Explain rationale for approach

82

Inspection Meeting Notice

Date:

From : Moderator

To: Inspection Team

Inspection Package: (Path Name)

Overview Meeting: (Date,time(From,To), Place)

Defect Logging Meeting: (Date,time(From,To),

Place)

83

  Moderator

◦  Ensure that the participants understand their roles

◦  Ensure that the author presents an overview

◦  Note time spend in Overview Meeting

  Author

◦  Present an overview of the work product

◦  Explain the operating environment, function & rationale for

approach

 Reader, Inspector

◦  Understand the Context

◦  Gets a clear picture of the operation of the work Product

85

  Moderator

◦  Learn the material to be Inspected

◦  Inspect the work product and prepare the Inspector Form

◦  Based on the Inspector from from Inspectors: Confirm that the

work product is ready for Inspection

  Ensure that the Inspectors have Invested required effort in

preparation

◦  Ensure that the reader is well prepared to paraphrase what the

work product docs

86

 Reader

◦  Understands the work product well to explain its working to

the Inspection team

◦  Divide the work product in to suitable portions for

paraphrasing

◦  Prepare a write-up to help you paraphrase the functionality of

each part of the work Product

◦  Inspect the work product & prepare Inspector Form

◦  Send Inspector Form to the moderator

 Inspectors

◦  Learn the material to be Inspected

◦  Inspect the work product and prepare Inspector Form

◦  Send Inspector form to the moderator

 Author

◦  Inspect the work product and prepare Inspector Form

◦  Ensure that the reader paraphrases the work product

◦  Ensure that the Inspection rate is not too fast or not too slow

◦  Ensure that the discussions are limited to making the author understand

the issue raised

◦  On detecting a defect stop the Inspection & get the defect recorded in

the Inspection Log

◦  Categorize the defect in to Major/Minor/Trivial/Investigate

91

 Investigate

◦  A condition that cannot be conclusively categorized as a

defect or not a defect based on the information

available in the inspection

◦  A condition that needs further discussions &

investigation outside the defect logging meeting

◦  Ensure that the team does not fix the defect in the meeting

◦  Ensure that the entire work product is covered by the reader

◦  Get a resolution from the Inspection Team

  Accept Unconditionally

  If the Inspection team feels that there is no need for

anyone to check the work product after the author

  Reject

  If the Inspection team feels the need for re-inspection of

the work product

94

  Accept Conditionally

◦  If the Inspection team feels the need for the sanity

checks for the defects fixes after rework by author

◦  If the work product is accepted conditionally , form

a closure action team (CAT)

◦  Inspect the work product not the author

◦  Ensure that the objective of any discussion in the

meeting is only to make the author understand the

issue

◦  Do not fix defects in the meeting

Defect Logging Meeting

Work Product Process Inputs

98

 Author

◦  Ensure that all defects are fixed

◦  Ensure that all Investigate Items are closed

◦  Ensure that all documents are properly updated

 Author

◦  Explain fixing of all defects to CAT

◦  Explain closure of each Investigate item to CAT

◦  Report updating of all relevant documents to CAT

◦  Ensure that all defects are fixed

◦  Ensure that all Investigate item are closed

◦  Ensure that all relevant documents are properly updated

100

 Moderator

◦  Report Inspection Metrics to QA/SEPG

◦  Report Process Improvement opportunities to the

action system

101

  Only code needs to be Inspected

◦  Inspection work for all written documents like requirements, project

plans, design, code, test plan, etc

  Inspections are suitable for only projects starting from the scratch

◦  Inspection are very effective in maintenance activities like

enhancements and bug fixing

  Inspections can be used only by large project teams

◦  Inspections are very useful in small project teams with external

Inspectors if needed

  Only experienced engineers are effective Inspectors

◦  Sometimes most intricate defects are found by new members

of the team

  We cannot implement Inspections because we do not have a well

defined software development process

◦  Inspections can be the starting point for defining the software

development process

103

  Improvement in Quality of deliverables

  Learning of good ideas & potential pitfalls

  Reduction of rework cost

  Continuous Process Improvement

106

effectiveness of Inspection

  Size of the artifact

  No of defects found

  No of Major Defects found

  If no of major defects found are very less compared to minor or trivial

defects, moderator can ask the Inspectors to re-do the preparation

  Defect Density of the Inspection session (Total no of weighted

defects found / size of the work product)

  Defect Detection Rate (no of Major defects found per hour of

Defect Logging meeting)

  Rate of review (No of LOC/pages covered per Hr of Inspection)

  Average Defect Density for different types of work products

  Average Rate of Review for different types of work products

107

 Review Effectiveness

◦  RE = No of Review Defects found

Total No of Defects found before delivery*

(* Both in Reviews and Testing during the entire life cycle)

◦  Computed on Weighted Defects

  Major, Minor, Trivial

  5,3,1 (suggested)

 

 GOAL 65% (minimum)

110

115

  Cooperation of the Author is key

  More defects can be found

  Knowledge sharing among team

  Mentoring of new reviewers

  Helped in finding bugs missed out in testing

  Advance preparation is mandatory

  Moderator’s role is very important

  Allocating time for reviews is a problem

- How to Institutionalize this best practice?

  MANAGEMENT SUPPORT!

  Pilot Program

o  Training

o  Selection of appropriate work products

o  Inspect the early Inspections !

o  External support (Optional)

  Ongoing Inspection program

o  Structure

o  Process definition

o  Set internal benchmarks

o  Constant monitoring