Test Logs - Introduction

1 Test Logs - Introduction Test Problem is a condition that exists within the software system that needs to be addressed.. As described the most common test Report is a simple Spread she

1 Test Logs - Introduction

Test Problem is a condition that exists within the software system that needs to be addressed

Carefully and completely documenting a test problem is the first step in correcting the problem

The following four attributes should be developed for all the test problems:

Statement of condition –Tells what it is

Criteria – Tells what should be

These two attributes are the basis for a finding If a comparison between the two gives little or no practical consequence, no finding exists

Effect: Tells why the difference between what is and what should be is significant

Cause: Tells the reasons for the deviation Identification of the cause is the necessary as a basis for corrective action

A well developed problem statement will include each of these atttributes.When one or more these attributes is missing, questions almost arise, such as

Criteria: Why is the current state inadequate?

Effect: How significant is it?

Cause: What could have cause of the problem?

Document Deviation:

Problem statements begin to emerge by process of comparision.Essentially the user compares” what is” with “what should be” When a deviation is identified between what is found to actually exist and what the user thinks is correct or proper , the first essential step toward development of a problem statement has occurred It is difficult to visualize any type of problem that is not in some way characterized by this deviation The ‘What is”: can be called the statement of condition The “What should be” shall be called the “Criteria” These concepts are the first two and the most basic , attributes of a problem statement The documenting of the deviation is describing the conditions, as they currently exist, and the criteria, which represents what the user desires

The actual deviation will be the difference or gap between “what –is” and “ what is desired”

The statement of condition is uncovering and documenting the facts, as they exist

What is a fact? The statement of condition will of course depend on the nature and extent of the

evidence or support that is examined and noted For those facts, making up the

statement of condition, the I/S professional will need to ensure that the information is accurate, well supported, and worded as clearly and precisely as possible

The statement of condition should document as many of the following attributes as appropriate of the problem

Activities Involved:- The specific business or administered activities that are being performed during Test Log generation are as follows:

Procedures used to perform work – The specific step-by –step activities that are utilized in producing the output from the identical activities

Outputs /Deliverables – The products that are produced from the activity

Inputs - The triggers,events,or documents that cause this activity to be executed

Users/Customers served –The organization ,individuvals,or class users/customers serviced by this activity

Deficiencies noted – The status of the results of executing this activity and any appropriate

interpretation of those facts

The Criterion is the user’s statement of what is desired It can be stated in the either negative or positive terms For example , it could indicate the need to reduce the complaints or delays as well as desired processing turn around time

Work Paper to describe the problem, and document the statement of condition and the statement of criteria For example the following Work paper provides the information for Test Log Documentation:

Field Requirements: Field Instructions for Entering Data

Name of Software Tested : Put the name of the S/W or subsystem tested.

Problem Description: Write a brief narrative description of the variance uncovered from expectations

Statement of Conditions: Put the results of actual processing that occurred here.

Statement of Criteria: Put what testers believe was the expected result from processing

Effect of Deviation: If this can be estimated , testers should indicate what they believe the impact or

effect of the problem will be on computer processing

Cause of Problem: The testers should indicate what they believe is the cause of the problem, if known.

If the testers re unable to do this , the work paper will be given to the development team and they should indicate the cause of the problem

Location of the Problem: The Tests should document where problem occurred as closely as possible Recommended Action: The testers should indicate any recommended action they believe would be

helpful to the project team If not approved, the alternate action should be listed or the reason for not following the recommended action should be documented

Name of the S/W tested:

Problem Description

Statement of Condition

Statement of Criteria

Effect of Deviation

Cause of a Problem

Location of the Problem

Recommended Action

Four categories of data will be collected during testing These are explained in the following paragraphs.

Test Results Data

This data will include,

Test factors -The factors incorporated in the plan, the validation of which becomes the Test Objective.

Business objective –The validation that specific business objectives have been met.

Interface Objectives-Validation that data/Objects can be correctly passed among Software components.

Functions/Sub functions-Identifiable Software components normally associated with the requirements of the software.

Units- The smallest identifiable software components

Platform- The hardware and Software environment in which the software system will operate.

Test Transactions, Test Suites, and Test Events

These are the test products produced by the test team to perform testing.

Test transactions/events: The type of tests that will be conducted during the execution of tests, which will be based on software requirements

Inspections – A verification of process deliverables against deliverable specifications

Reviews: Verification that the process deliverables / phases are meeting the user’s true needs

Defect

This category includes a Description of the individual defects uncovered during the testing process This description includes but not limited to :

Data the defect uncovered

Name of the Defect

Location of the Defect

Severity of the Defect

Type of Defect

How the defect was uncovered(Test Data/Test Script)

The Test Logs should add to this information in the form of where the defect originated , when it was corrected, and when it was entered for retest.

Storing Data Collected during Testing

It is recommended that a database be established in which to store the results collected during testing

It is also suggested that the database be put in online through client/server systems so that with a vested interest in the status of the project can be readily accessed for the status update

As described the most common test Report is a simple Spread sheet , which indicates the project component for which the status is requested, the test that will be performed to determine the status of that component, and the results of testing at any point of time

Developing Test Status Reports

Report Software Status

Establish a Measurement Team

Inventory Existing Project Measures

Develop a Consistent Set of Project metrics

Define Process Requirements

Develop and Implement the Process

Monitor the Process

The Test process should produce a continuous series of reports that describe the status of testing The test reports are for use of testers, test managers, and the software development team The frequency of the test reports should be based on the discretion of the team and extensiveness of the test process

Use of Function/Test matrix:

This shows which tests must be performed in order to validate the functions and also used to determine the status of testing Many organizations use spreadsheet package to maintain test results The

intersection can be coded with a number or symbol to indicate the following:

1=Test is needed, but not performed

2=Test currently being performed

3=MINOR DEFECT NOTED

4=Major defect noted

5=Test complete and function is defect free for the criteria included in this test

TEST

FUNCTION 1 2 3 4 5 6 7 8 9

A

B

C

D

E

Function Test Matrix

1 Methods of Test Reporting

Reporting Tools - Use of word processing, database, defect tracking, and graphic tools to prepare test

reports

Some Database test tools like Data Vision is a database reporting tool similar to Crystal Reports Reports can be viewed and printed from the application or output as HTML, LaTeX2e, XML, DocBook,

or tab- or comma-separated text files From the LaTeX2e and DocBook output files you can in turn produce PDF, text, HTML, PostScript, and more Some query tools available for Linux-based databases include:

QMySQL

dbMetrix

PgAccess

Cognos Powerhouse

This is not yet available for Linux; Cognos is looking into what interest people have in the product to assess what their strategy should be with respect to the Linux ``market.''

GRG - GNU Report Generator

The GRG program reads record and field information from a dBase3+ file, delimited ASCII text file or a SQL query to a RDBMS and produces a report listing The program was loosely designed to produce TeX/LaTeX formatted output, but plain ASCII text, troff, PostScript, HTML or any other kind of ASCII based output format can be produced just as easily

Word –Processing:

One way of increasing the utility of computers and word processors for the teaching of writing may be to use software that will guide the processes of generating, organizing, composing and revising text This allows each person to use the normal functions of the computer keyboard that are common to all word

processors, email editors, order entry systems, and data base management products From the Report Manager, however, you can quickly scan through any number of these reports and see how each person's history compares A one-page summary report may be printed with either the Report Manager program or from the individual keyboard or keypad software at any time Individual Reports include all of the following information

Status Report

Word Processing Tests or Keypad Tests

Basic Skills Tests or Data Entry Tests

Progress Graph

Game Scores

Test Report for each test

Test Director:

 Facilitates consistent and repetitive testing process

 Central repository for all testing assets facilitates the adoption of a more consistent testing process, which can be repeated throughout the application life cycle

 Provides Analysis and Decision Support

 Graphs and reports help analyze application readiness at any point in the testing process

 Requirements coverage, run schedules, test execution progress, defect statistics can

be used for production planning

 Provides Anytime, Anywhere access to Test Assets

 Using Test Director’s web interface, tester, developers, business analysts and Client can participate and contribute to the testing process

 Traceability throughout the testing process

 Test Cases can be mapped to requirements providing adequate visibility over the test coverage of requirements

 Test Director links requirements to test cases and test cases to defects

 Manages Both Manual and Automated Testing

 Test Director can manage both manual and automated tests (Win Runner)

 Scheduling of automated tests can be effectively done using Test Director

Test Report Standards - Defining the components that should be included in a test report

Statistical Analysis - Ability to draw statistically valid conclusions from quantitative test results.

Testing Data used for metrics

Testers are typically responsible for reporting their test status at regular intervals The

following measurements generated during testing are applicable:

Total number of tests

Number of Tests executed to date

Number of tests executed successfully to date

Data concerning software defects include

Total number of defects corrected in each activity

Total number of defects entered in each activity.

Average duration between defect detection and defect correction

Average effort to correct a defect

Total number of defects remaining at delivery

Software performance data us usually generated during system testing, once the software has been integrated and functional testing is complete.

Average CPU utilization

Average memory Utilization

Measured I/O transaction rate

Test Reporting

A final test report should be prepared at the conclusion of each test activity This includes the following

 Individual Project Test Report

 Integration Test Report

 System Test Report

 Acceptance test Report

These test reports are designed to document the results of testing as defined in the testplan.The test report can be a combination of electronic data and hard copy For example, if the function matrix is maintained electronically, there is no reason to print that, as paper report will summarize the data, draws appropriate conclusions and present recommendations.9 - Purpose of a Test Report:

The test report has one immediate and three long term purposes The immediate purpose is to provide information to customers of the software system so that they can determine whether the system is ready for production , and if so, to assess the potential consequences and initiate appropriate actions to minimize those consequences

The first of the three long term uses is for the project to trace problems in the event the application malfunctions in production Knowing which functions have been correctly tested and which ones still contain defects can assist in taking corrective actions

The second long term purpose is to use the data to analyze the rework process for making changes to prevent the defects from occurring in the future These defect prone components identify tasks/steps that if improved, could eliminate or minimize the occurrence of high frequency defects The Third long term purpose is to show what was accomplished in case of an Y2K lawsuit

 Individual Project Test Report

These reports focus on the Individual projects(software system),when different testers should test individual projects, they should prepare a report on their results

 Integration Test Report

Integration testing tests the interfaces between individual projects A good test plan will identify the interfaces and institute test conditions that will validate interfaces Given is the Individual Project test report except that conditions tested are interfaces

1.Scope of Test – This section indicates which functions were and were not tested

2.Test Results – This section indicates the results of testing, including any variance between what

is and what should be

3.What works/What does not work - This section defines the functions that work and do not work and the interfaces that work and do not work

4 Recommendations – This section recommends actions that should be taken to

Fix functions /Interfaces that do not work

Make additional improvements

 System Test Reports

A System Test plan standard that identified the objective of testing , what was to be tested, how was it to

be tested, and when tests should occur The system test Report should present the results of executing the test plan If these details are maintained Electronically , then it need only be referenced , not included in the report

 Acceptance Test Report

There are two primary objectives of Acceptance testing Report The first is to ensure that the system as implemented meets the real operating needs of the user/customer If the defined requirements are those true needs, testing should have accomplished this objective

The second objective is to ensure that software system can operate in the real world user environment, which includes people skills and attitudes, time pressures, changing business conditions, and so forth The Acceptance Test Report should encompass these criteria’s for the User acceptance respectively

2 Conclusion

The Test Logs obtained from the execution of the test results and finally the test reports should be designed to accomplish the following objectives:

 Provide Information to the customer whether the system should be placed into production, if so the potential consequences and appropriate actions to minimize these consequences

 One Long term objective is for the Project and the other is for the information technology function

 The project can use the test report to trace problems in the event the application malfunction in production Knowing which functions have been correctly tested and which ones still contain defects can assist in taking corrective actions

 The data can also be used to analyze the developmental process to make changes to prevent defects from occurring in the future

 These defect prone components identify tasks/steps that if improved, could eliminate or minimize the occurrence of high frequency defects in future