Customer Satisfaction Report

Our objective is to understand exactly how our customers perceive the Institute and, based on their level of satisfaction, to improve our services and products.. The customer segments a

EUROCONTROL

A Centre of Excellence in ATM Training

IANS

Customer Satisfaction

Report

Copyright Notice

© 2006 European Organisation for the Safety of Air Navigation (EUROCONTROL) All rights reserved

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise

Any other use is subject to prior written consent by EUROCONTROL Request shall be addressed to:

Director of the Institute of Air Navigation Services, 12, rue Antoine de Saint-Exupéry, L-1432 Kirchberg/Luxembourg

Table of Contents

INTRODUCTION 4

ATM TRAINING 5

ATC TRAINING 12

E-LEARNING 13

STAKEHOLDER FEEDBACK 14

COMPLAINTS 15

CONCLUSION 16

This is the 7th report that the Institute devotes to the

assessment of the level of satisfaction of its

customers The study on which this report is based

has been conducted in the framework of the quality

improvement programme implemented at the

Institute since end of 1996

Measuring customer satisfaction on a regular basis

is one of the key elements of the quality strategy

implemented by the Institute since 1997

Our objective is to understand exactly how our

customers perceive the Institute and, based on their

level of satisfaction, to improve our services and

products

This report proposes a summary and an analysis of

the data collected in 2005

The customer segments

assessed in 2005

IANS applies a policy to measure the satisfaction of

its customers with all its products and services

These include:

• The ATM course participants

• The ATC course participants

• The E-Learning Students

• Training Sponsors

Assessment Tools

The assessment is based on a set of assessment

tools and measurement methods During 2005

further work has been done to increase the integrity

of the data collection

Evaluation Forms

To measure the satisfaction of its customers with

the products and services IANS uses the following

evaluation forms:

• A Student Evaluation Form filled in by the students at the end of a course This evaluation form is used for both the ATM course participants and ATC course participants

• An E-Learning Evaluation Form filled in by a student upon completion of an E-Learning Module

• A generic Stakeholder Feedback Form filled in

by various stakeholders The form is available

at http://www.eurocontrol.int/ians/feedback.html

Customer Visits

IANS visits a sample of its customers on a yearly basis The customer visits serve as a main source

of feedback on existing products and a source of information for improvements and new products

The customer visits include visits to Air Navigation Service Providers and civil and military State Authorities

Complaint Procedure and Complaint Form

IANS uses an anonymous form, available on its web-site for the collection of complaints related to its products, services and staff A reference to the complaint procedure and associated form is made

in all products and services The complaint form is available at http://www.eurocontrol.int/ians/complaint.html

ATM Training

The Satisfaction of Students with the ATM Training Products is measured through the use of a Course Evaluation

Form

Measurement

Method

At the end of each

(classroom) course,

students are presented

with an evaluation form

containing a number of

criteria to be assessed

Each criterion can be

given a result between 1

and 4 meaning

unacceptable (1),

unsatisfactory (2), good

(3) an excellent (4)

The complete set of

criteria are grouped in

five main categories:

• Instructional Team

• Documentation

• Equipment

• Administration

The first three categories

are directly related to the

course The last two

categories are directly

related to the supporting

environment and

organisation

Figure 1 Student Evaluation Form

From Forms to

Results

We map the marks of the

students on a scale from

0 (all students mark

Unacceptable) to +100

(all students mark

Excellent) When all

students mark Good

then the result is 75 and

when students mark, on

average, between Good

and Excellent then the

result will be around 85

Likewise, if students fill

in 'all-Uncacceptable'

then the result will be 0

If students fill in

'all-Un-Satisfactory' then the

result will be 25

1

©Copyright 2003 EUROCONTROL

Measuring Satisfaction

1

Unsatisfactory Satisfactory

2

Good

3

Excellent

4

Marks from Students

Result

All 3 All 2

All 1

0

All 4

Figure 2 How we measure satisfaction

IANS aims to meet our objective of being a Centre

of Excellence in ATM Training For us this means

we aim to achieve results on any criterion between

75 and 100 As an example: the overall average (all

courses, all criteria, 3090 forms) for IANS in 2005 is

87.5 – compared to 85 in 2004 The highest result

on a single criterion is 95 (Subject Knowledge of the

Instructor) The lowest result remains pre-course

information (76.5 – compared to 70 in 2004)

Results

The results presented below are extracted from the automated student satisfaction evaluation system Results can be presented per category, per training domain and per criterion We have selected the most significant results to be included in this report

0 500 1000 1500 2000 2500 3000 3500

Figure 3 Number of Evaluation Forms

Size of the Population

This graph shows the number of evaluation forms received and processed by IANS In

2005 the Institute continued to optimise the processing of the forms

This led to another significant increase of the number of processed forms in 2005

Composition of

the Population

Students indicate on the

evaluation form which

type of staff they are

This chart shows that a

majority of Students at

IANS in 2005 are

Operational and

Technical Staff

These results are

consistent with the

results of the previous

years

192

345

977

693

223

600

1298

824

0 200 400 600 800 1000 1200 1400

Administrative Managerial Operational Technical

2004 2005

Figure 4 Composition of the population (number of students)

50 55 60 65 70 75 80 85 90 95 100

Figure 5 ATM Course Results

Satisfaction

This chart shows the satisfaction of the students with the ATM courses The result is obtained by averaging the results obtained on every criterion in the categories Training Content, Instructional Team and

Documentation

A result of 87 indicates that, on average, students mark criteria between Good and Excellent

The increase in satisfaction has been achieved through detailed analysis of the results of each course and introducing corrective action wherever required

50 55 60 65 70 75 80 85 90 95 100

Figure 6 ATM Course Results: per Category

Segmented View

This chart presents the results per measurement Category These are Training Content, Instructional Team and Documentation

IANS has improved the results on each of these three categories during the last four years IANS will continue to analyse the results and make improvements where possible and practicable

‘Best’ Results

Customers often praise

IANS for its staff and

excellent infrastructure

This informal feedback is

confirmed by the course

evaluation results The

criteria ‘Instructor

Subject Knowledge’ and

‘Classroom Equipment’

have consistently

received the highest

marks from the students

during the last three

years

IANS will continue to

invest in the competence

of its instructors by

continuation training and

competency assessment

and it will maintain the

building at its high

standard

50 60 70 80 90 100

Instructor Subject Knowledge

Classroom Equipment

Figure 7 Criteria with ‘Best’ Results

50 60 70 80 90 100

Pre Course

Information

Figure 8 Criteria with less positive Results

Areas for Improvement

IANS continuously looks for opportunities to improve Criteria with the less positive results are the Pre-Course Information and the Course Duration

In 2004 IANS has put in place an action plan to improve the Pre-Course information and significant progress has been made in this domain in 2004 and

2005

The optimum duration for each course will be reviewed on a course-by-course basis

In 2005 IANS changed the layout of the coffee

Focus on Safety

IANS is heavily involved

in the development and

delivery of Safety

Courses to support

Safety Regulation and

Safety Management

In 2005 the number of

courses delivered in the

Safety domain increased

from 36 to 50 and the

number of participants

increased from 614 to

844

We are pleased to note

that the results in this

relatively new area for

IANS equal the average

obtained in other, more

mature, training

domains

It is a clear indication

that the rigorous

development approach

at IANS leads to

products that meet

customer expectation

50 55 60 65 70 75 80 85 90 95 100

Figure 9 Safety Domain Course Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Figure 10 Would recommend course to a colleague

Measuring Loyalty

In 2004 we introduced a new question on the evaluation form: ‘Would you recommend the course to a colleague’

In 2005, like in 2004, more than 90% of the students would recommend the course that they attended to a colleague

1,00%

2,00%

3,00%

4,00%

5,00%

6,00%

7,00%

8,00%

9,00%

10,00%

Percentage of

'Unsatisfactory'

Results

Figure 11 Level of dissatisfaction

Measuring dissatisfaction

To measure dissatisfaction IANS uses the percentage of

‘unsatisfactory’ marks on the evaluation forms

This chart shows the proportion of un-satisfactory marks on the evaluation forms for all criteria

The excellent results in

2005 are spread over the entire spectrum of criteria

Note that that an

‘unsatisfactory’ mark on one criterion does not necessarily mean that the student is not satisfied with the whole product

ATC Training

The satisfaction of ATC ab-initio students is

measured using the same evaluation as the system

used for measuring the satisfaction of the ATM

students Satisfaction figures of ATM and ATC

students must however not be compared The

context of these courses, although delivered in the

same building, is completely different:

• The duration of ATM courses is maximum of

two weeks whereas ATC have a duration of

several months

• The student population of ATM courses are

staff members from service providers and

member states

• The student population of ATC courses are

ab-inito ATC students

• Last but not least, the ATC ab-initio students are evaluated at the end of the course At the end of the evaluation they either pass or fail The outcome of the evaluation tends to have a significant impact on the satisfaction of the students

IANS believes that the critical success factor for ATC students is their validation rate (i.e the proportion of students that obtain an ATC license)

We do however present the satisfaction data in this report as it is part of our quality management tools

50 60 70 80 90 100

Training

Documentation

Figure 12 ATC Student Satisfaction

ATC Student Satisfaction

This chart shows the satisfaction of ATC ab-initio students

The last few years ATC ab-initio training has been characterised by high rotation of staff and continuous changes to the training content and structure as a result of variations in demand and new regulatory

requirements

In 2005, IANS significantly improved the

documentation and training of the ATC Instructors

Compared to 2004 the satisfaction of the students has increased for each segment

E-Learning

IANS introduced satisfaction measurement for its

e-learning products in the middle of 2004 Students

are given an opportunity to evaluate each e-learning

module at the end of the module The figures

presented in this report cover all the modules on the

EUROCONTROL IANS on-line TrainingZone,

including the CFMU modules

In 2005, students filled in 2774 forms (800 in 2004) evaluation forms were filled in 90% of the students identified to be ‘satisfied’ with the modules and more than 88% of the students indicated that the module met the objectives

These figures indicate that the overall satisfaction with the e-learning modules is excellent

1 Why are you studying this module?

‰ Personal reasons (e.g to further career, out of personal interest etc)

‰ My employer decided I should

‰ It was a prerequisite for a class-room based course or programme

4 In order to successfully access this module, did you have to

‰ change your browser (Internet Explorer) settings?

‰ install new software (flash plug-in, java plug-in, internet explorer)?

‰ disable a popup-killer or anti-intrusion programme (e.g Norton Internet

Security)?

7 Will this module

‰ Help you directly in your current job?

‰ Help you in a future job?

‰ Increase your understanding of the industry without direct impact on your

current/future job?

‰ Help you personally (e.g out of personal interest)?

E-Learning Evaluation Form

A customised form was developed to capture student satisfaction with the e-learning modules

The form collects information related to the

‘reasons for studying’, the complexity of the module, the objectives of the module and the impact of the module on the student’s job

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Structure of E-Learning Modules

This chart shows the percentage of students that answered ‘yes’ on the following questions:

• Structure of the module meets expectations

• Level of the module meets the expectations

5%

10%

15%

20%

25%

30%

35%

40%

45%

Figure 14 Why did students study e-learning modules

Origin of E-Learning Students

It is IANS’ strategy to integrate e-learning in well defined training programmes – defined and implemented by the student’s employer - and

as pre-requisite modules for classroom courses These results indicate that the proportion of students registered through their employer or studying as a pre-requisite module has significantly increased compared to the number

of students that study out

of their own initiative The number of students that see a direct impact

on their job from studying e-learning modules has increased from 56% in 2004 to 68% in 2005

Stakeholder Feedback

Training Consultation Group

The Stakeholder Feedback Form was distributed to

the meeting participants at the end of each Training

Consultation Group Meeting The majority of the

feedback indicates that the overall perception of the

Institute lies between Good and Excellent (on a

scale unsatisfactory – satisfactory – good –

excellent)

Compared to 2004 the results related to

communication (content, format and timeliness)

have improved We will continue to review and

improve our communication with the members of the consultation group

Other stakeholders

The stakeholder evaluation form was also distributed to:

• Participants of the training focus group

• The training partners at IATA

The results indicate an equally high level of satisfaction of working together with IANS

Customer Visits

The Customer Visits are a process in which IANS

meets stakeholders in member states on a bi-lateral

basis It is a mechanism to collect qualitative

feedback from various stakeholders The countries

visited in 2005 are: Austria, Belgium, BIH, Bulgaria,

Greece, Hungary, Spain, Sweden and Turkey

During the meeting of the Training Consultation

Group in September 2005 it was agreed that IANS

would include in its Customer Satisfaction Report

information gathered during the Customer Visits

During these customer visits, stakeholders have in

general expressed a high degree of satisfaction with

IANS, its staff and the services it offers The

following points have been raised:

• OJTI: The delivery of OJTI courses through

partnerships is often perceived as an

expensive solution (by stakeholders receiving

the services) The solution is not perceived as

entirely satisfactory

• ATSEP Training: Many of the visited stakeholders are currently in the process of defining basic training for Air Traffic Safety Electronics Personnel (ATSEP) Some collaboration with EUROCONTROL and other member states could avoid duplication of effort

• On-Site Courses: In general the demand for on-site course (i.e courses at stakeholder premises) exceeds the capacity of IANS

• Although most stakeholders are aware of the E-Learning products of the Institute many of them have not yet been able to assess the feasibility to use the products available

• ATC Training Notes: IANS should develop a policy to distribute CCC aligned ATC Training Notes

Complaints

Every training note distributed to students and all

pages on the On-Line Trainingzone contain a

clearly visible reference to IANS’ complaint

procedure The procedure enables stakeholders to

submit complaints in an anonymous way or an

identified complaint through e-mail The text is as

follows:

The EUROCONTROL Institute of Air Navigation

compliment) please tell us If you are not satisfied with the service we provide or you would like to propose an improvement then please fill out the form at http://www.eurocontrol.int/ians/complaint.html, or

contact IANS.complaints@eurocontrol.int directly

Although the procedure is clearly marked in all