LV Thạc sỹ_Improve information system supporting monitoring and evaluation of greater mekong subregion sustainable tourism development project in Viet Nam

Experience in application of Information system supporting M&E in Viet Nam Land Administration Project VLAP funded by the World Bank...38 CHAPTER 3: CURRENT SITUATION OF INFORMATION SYST

The Final Thesis on “Improving Information System supporting Monitoring andEvaluation (M&E) of Greater Mekong Subregion Sustainable TourismDevelopment Project in Viet Nam” has received valuable assistance andcooperation from many people, whom I am sincerely grateful for The first personthat I would like to express my gratitude is Dr …

I also would like to thank the respondents from the Project Implementation Unit(PIUs) at the provincial, district and communal levels and other stakeholders:ADB’s assistants, the GoV and related agencies’ officers for their extensiveknowledge and experience shared as well as their concerns on issues of M&EInformation System

TABLE OF CONTENTS

ABBREVIATION 5

LIST OF FIGURES 7

EXECUTIVE SUMMARY 8

INTRODUCTION 10

1 RATIONALE 10

2 RESEARCH OBJECTIVES: 11

3 RESEARCH QUESTIONS: 12

4 RESEARCH METHODOLOGY 12

4.1 Research phases 12

4.2 Methods of data collection 13

5 RESEARCH SCOPE 15

6 THESIS STRUCTURE 15

CHAPTER 1: THEORETICAL BACKGROUND ON INFORMATION SYSTEM (IS) AND INFORMATION SYSTEM DEVELOPMENT METHODS 16

1.1 Theoretical background on Information System 16

1.1.1 Definition of Information System 16

1.1.2 Information system perspectives 16

1.1.3 Functions of IS in organization 18

1.1.4 Impacts of information systems on organizations 19

1.2 Information system development methods 20

1.3 Criteria to evaluate an Information System 23

CHAPTER 2: MONITORING AND EVALUATION (M&E) AND INFORMATION SYSTEM (IS) SUPPORTING M&E IN ODA PROJECTS 27

2.1 Overview of Monitoring & Evaluation 27

2.1.1 Definitions of M&E 27

2.1.2 Monitoring & Evaluation classification 30

2.1.3 Role of M&E in project implementation 31

2.1.4 General regulations on ODA project M&E 32

2.1.5 M&E steps 32

2.2 The supportive relationship between IS and M&E 33

2.3 Principles to build an information system supporting Monitoring & Evaluation 35

2.4 Experience in applications of Information system supporting M&E in some ODA projects in Viet Nam 35

2.4.1 Experience in application of Information system supporting M&E in Road Network Improvement Project (RNIP) funded by the World Bank 36

2.4.2 Experience in application of Information system supporting M&E in Viet Nam Land Administration Project (VLAP) funded by the World Bank 38

CHAPTER 3: CURRENT SITUATION OF INFORMATION SYSTEM SUPPORTING MONITORING &EVALUATION IN STDPVN 41

3.1 STDPVN introduction 41

3.1.1 The overview of project 41

3.1.2 Project Organizational Chart 42

3.2 Monitoring & Evaluation in the Project 43

3.2.1 M&E system 43

3.2.2 Reporting arrangement 45

3.3 Description of STDPVN’s IS supporting M&E 45

3.3.1 Overview of STPDVN’s IS supporting M&E 45

3.3.2 Description of STDPVN’s IS supporting M&E 47

3.4 Assessment on the current STDPVN’s IS supporting M&E 55

3.4.1 Level of use 55

3.4.2 Users’ satisfaction 56

3.4.3 Attitudes of users toward the current information system 64

3.4.4 The extent to which current IS meet specified objectives 65

3.4.5 Financial payoff in application of the information system 66

CHAPTER 4: RECOMMENDATIOINS FOR IMPROVING INFORMATION

SYSTEM SUPPORTING MONITORING AND EVALUATION IN STDPVN 68

4.1 Gaps between key requirements and current situation of IS 68

4.2 Recommendations for improving IS supporting M&E 70

4.3 Implementation plan to improve STDPVN’s IS supporting M&E: 80

4.4 Budget for improving IS supporting M&E 83

CONCLUSION 85

BIOGRAPHIC REFERENCES 86 APPENDICES

Appendix 1: Survey questionnaires

Appendix 2: List of respondents

Appendix 3: Sample key in-depth Interviews guide

Appendix 4: List of interviewees

Appendix 5: RNIP’s AMT formats

Appendix 6 Summary of VLAP – M&E - IS solution

Appendix 7: Key elements of RFP to the IS supporting M&E in STDPVN Appendix 8: STDPVN’s IS output – Weighted progress

MCTO Mekong Tourism Coordination Office

NPSC National Project Steering Committee

ODA Official Development Assistance

PCU Project Coordination Unit

PIU Project Implementation Unit

PPC Provincial People’ Committee

PPSC Provincial Project Steering Committee

SPSC Sub regional Project steering Committee

STDPVN Greater Mekong Subregion Sustainable Tourism Development

Project in Viet NamMCST Ministry of Culture, Sports and Tourism

M&E Monitoring & Evaluation

MPI Ministry of Planning and Investment

MBA Master of Business Administration

VLAP Viet Nam Land Administration Project

LIST OF TABLES

Table 1 Positive and negative impacts of information systems 20

Table 2 Comparison of Systems-Development Methods 23

Table 3 Differences between Monitoring and Evaluation 28

Table 4 M&E classification 31

Table 5 STDPVN’s reporting arrangement 45

Table 6 Frequency of use of main information sharing methods 55

Table 7 Level of using various tools and technologies 56

Table 8 Perception of respondents on IS contents’ quality 57

Table 9 Respondents’ assessment on training course 62

Table 10 STDPVN’s IS SWOT analysis 66

Table 11 Gaps between key requirements and current IS situation 68

Table 12 Summary of recommendations for improving STDPVN’s IS 80

Table 13 Action plan to improve STDPVN’s IS 81

Table 14 Cost estimate for building and installing STDPVN_M&E_IS 84

LIST OF FIGURES

Figure 1 Research phases 12

Figure 2 The MTO model - Information system perspective 17

Figure 3 The IPO model - Functions of IS in organization 19

Figure 4 “Operation area” of Monitoring and Evaluation 29

Figure 5 M&E data process 30

Figure 6 The model of supportive relationship between IS and M&E 34

Figure 7 Project Organizational Chart 42

Figure 8 Summary of STDPVN’ IS functions supporting PPMS 46

Figure 9 Input sheets of STDPVN’ IS 48

Figure 10 Frequency of use of each storage method 52

Figure 11 IS flow from PIUs to PCU 54

Figure 12 Users’ satisfaction levels to IS phases and factors 57

Figure 13 Chart of information updating level of the current IS 60

Figure 14 Survey on the organized training courses 61

Figure 15 Respondents’ ideas for IS improvement 65

Figure 16 Support level of IS to M&E 65

Under ADB and the Government (GoV)’s regulations, the project has set up aProject Performance Monitoring System (PPMS) including Information System (IS)building However, the IS supporting PPMS has not effectively functioned causingdelay in project implementation.

The main objective of this research is to assess the current situation of IS supportingM&E activities in STDPVN Specifically, to determine the problems andrecommend solutions for improving it in reference with the experience in successfulapplication of IS in another ODA projects

To conduct the research, the author has combined the methods of desk research,observation, in-depth interview to 12 people who are almost leaders, managers, andM&E staffs in STDPVN and other projects which have applied the IS supportingM&E as well as survey 30 STPDN’s officers The followings are main findings:

- Information produced by the current IS are inconsistent, inaccurate and unexhausted;

- The current IS has too many sheets with complicated formats and too many items in each sheet leading to difficulties in extracting needed information;

- Untimely information aggregation and slowly information updating/disseminating;

- Unskillful M&E staff and unclear cut responsibilities and functions of people involved;

- The project has lacked of appropriate technology: No full internal network, not standardized software used for M&E, limited storage volume and no good anti- virus system

The reasons for above weaknesses are

- Low managers’ awareness on the role of IS in supporting M&E at all levelscausing the loose coordination among units/departments in IS operation and IShas not been set up since the project inception;

- Lack of appropriate investment in IT leading to irrelevant and unsustainable ISapplied;

- Not good M&E staffing, lack of relevant training courses, lack of mechanism to

motivate M&E staff, almost M&E staff work in pluralism

The followings are the author’s recommendation to improve IS supporting M&E inSTDPVN:

- Refresh and strengthen hardware to serve the current IS’ s improvements;

- Setting up the internal network (LAN/WAN) to all PCU, PIUs;

- Selecting relevant method to develop the IS - Application Software Package toimprove the technology infrastructure: large database system; Frequent andautomatic storage and backup; Using firewall or some proper anti-virusprogram for secure

Organization

- Good M&E staffing at all levels: Professional M&E staff at all levels forseparate tasks of M&E; Organizing relevant training courses; Creating suitablepolicies to motivate M&E staff;

- Building a well-organized system with clear functions and responsibilities ofpeople involved

1 RATIONALE

Official Development Assistance (ODA) is one of official funds from developedcountries to developing countries with aim at supporting the economic developmentand social welfare for these countries Over the last 15 years, Viet Nam hassignificantly mobilized all resources and taking advantages of this fund for socio-economic development, poverty reduction and hunger mitigation However, due tolimited management capacity, bureaucratic mechanism, and corruption, projects areoften stuck with challenges, and not really effectively operated Delays in ODAproject progress are the main challenge for Viet Nam to reach the target timely.According to the World Bank (WB), general project progress is often slower thanexpected plan about 3 years Disbursement rate of Viet Nam is also considered thelowest in the region

Outstanding reason for limited management capacity in project is the lack of goodoperation of monitoring and evaluation (M&E) In addition to that, one of the mostimportant factors affecting effectiveness of M&E is the support of InformationSystem (IS) In project management, M&E plays a determined role to success ofproject It is highly required to consistently and effectively do those activities underthe Donor and the Government’s requirements, however awareness of IS in projectmanagement and the interrelation between IS and M&E is inadequate causingineffectiveness of M&E, and making delays in project management This is also thecase of Greater Mekong Subregion Sustainable Tourism Development Project(STDPVN)

STDPVN is financed by Asian Development Bank (ADB) with total amount of US

$12 million toward sustainable development and contribution to eco-sociodevelopment; especially focusing on hunger elimination and poverty reduction,tourism infrastructure development…It has 5 components carried out in 5 provinces(CIO Bang, Bac Kan, Quang Binh, Quang Tri, Hue) The project lasts in a period of

5 years (2009-2013) and is now being in the early of second year A lot ofstakeholders involved in project implementation such as National Steering

Committee - Ministry of Culture, Sports and Tourism (Executing Agencies), ProjectCoordination Unit and five Provincial Implementing Unit (PIUs), beneficiaries,tourism community groups

In term of project M&E, STDPVN must follow the requirements of both ADB andthe Government of Viet Nam A Project Performance Monitoring System (PPMS)had been systematically set up with M&E purposes: IMPLEMENTATION,COMPLIANCE AND IMPACTS M&E However, the Project does not have a clearand proper IS for effectively supporting the PPMS (M&E system) Problemshappen in almost phases of information management The IS is a simple excel filewhich supply a manual way to aggregate information instead of an automatic way.Accordingly, information processing also takes much time and the storage takesmuch space It has been found difficult to collect, analyze and manage informationcausing inconsistence in project management and implementation from the central

to provincial levels Therefore, M&E operation has also not well operate underADB and Government’s requirements That is main reason which creates delays inproject progress Currently, progress rate only reaches 14% compared with that ininitial plan (Aide Memoire, ADB Loan Review Mission, 5.2010)

In that context, basing on the theory of IS, M&E in ODA project and in referencewith some successful applications of IS in another projects, I decided to undertake aresearch on IS supporting M&E in STDPVN with aim at reviewing the realsituation of STDPVN’s IS and find out the gaps between the requirements and thereal status A better IS will be suggested to be applied for STDPVN

2 RESEARCH OBJECTIVES:

The thesis focused on the findings of current STDPVN’s IS and proposing solutionsfor improving the IS supporting M&E in STDPVN Hereafter are the specificobjectives:

- To review the theories of IS, M&E IS in ODA project

- To assess the current situation of IS supporting M&E activities in STDP VietNam to find out the gaps between the expected requirements and the actualsituation

- To propose recommendations to improve the current IS for better supportingthe M & E of STDP Viet Nam

3 RESEARCH QUESTIONS:

Basing on the objectives, the thesis has been developed to answer the questions:

- What theories of IS, IS supporting M&E in ODA project can be applied tosolve the selected research problem?

- What are the requirements for IS supporting M & E in STDPVN?

- What is the state of IS supporting M & E in STDPVN?

- What is the gap between requirements and state of IS supporting M & E inSTDPVN?

- What are solutions to improve IS supporting M & E activities in STDPVN?

4 RESEARCH METHODOLOGY

4.1 Research phases

Three main duties focused on the thesis are the theoretical background of IS,M&E, the relationship between IS and M&E in ODA project, the findings on thecurrent STDPVN’s IS supporting M&E and the solutions for improving the ISfor better supporting M&E in STDPVN Therefore a careful research plan wasprepared with milestones and specific activities to be done The thesis has beencarried out in three specific phases as described in the following figure

Source: Author’s analysis

Figure 1 Research phases

All phases were conducted using various data collection methods Thesecondary data on theoretical background and experience have been collected

Phase 3 – Current IS

assesment and solution for improvement

based on desk research method Data from in-depth interview, surveyquestionnaire and observation are used to evaluate current situation of IS andrecommendations are proposed accordingly.

4.2 Methods of data collection

Desk research

The following data sources have been collected:

- The IS textbooks from the training course of EMBA

- Documents of guidelines and regulations on ODA project M&E by theMinistry of Investment and Planning (MPI)

- Documents on M&E system from the sponsors, especially ADB

- STDPVN’s materials, especially M&E system

- Relevant Websites (ADB, MPI, MOF, MARD, MCST…)

- Some ODA projects’ IS materials

- Other related newspapers, magazines

This data collection helps to make clear about:

- Definitions on IS, , ODA M&E and the supportive role IS to M&E

- IS development methods in general and in STDPVN in particular

- Good experience in IS application in other ODA projects

- M&E system in STDPVN and a part of STPDPVN’s IS

Observation

In this research, observation method is employed to help answer the followingquestions:

- How is the information flow?

- How do the IS functions operate?

- How is M&E staffing in STDPVN?

- Whether the current IS is a well-organized system to support M&E?

- What does the IS infrastructure in STDPVN look like?

- How do STDPVN’s officers exchange or share information together or torelated agencies?

- What are the attitudes of users and stakeholders?

In-depth interviews

In-depth Interviews has been carried out with focus on three main groups (managers

at all levels, key M&E specialists and staff and people who directly related to ISperformance supporting M&E) Total number of interviewees is 12 with detailed

question sample in Table 1 and list of interviewees in Appendix 3 and 4 The

purpose of interviews is to get their opinions on:

- Requirements for IS supporting M&E under managerial, organizational andtechnical perspectives

- IS performance and assessment (including difficulties and advantages)

- STDPVN objective and direction

- Interviewees’ suggestions to improve the current system

The author also did the interviews to the managers and M&E staff in some otherproject which applied IS supporting M&E to note down:

- Their current IS performance and finding out what will be the best practicesfrom each system that can be suitable to improve the STDPVN’s IS?

- Their suggestions on improving the STDPVN’s IS

Survey

Total people for the survey are 30 with detailed survey questionnaire and list of

respondents in Appendix 1 and 2 This method also focuses on the followingissues:

- IS performance and assessment (including IS functions, difficulties andadvantages in IS performance)

- Their suggestions to improve the current IS

The feedbacks are very useful for the author to have exhaustive and comprehensiveinformation to the realistic and qualified thesis

5 RESEARCH SCOPE

The research scope is as follows:

- Research on IS supporting M&E system in STDPVN funded by ADB (underADB standard of M&E)

- Project areas: PCU (Ha No) and 05 PIUs (Cao Bang, Bac Kan, Quang Binh,Quang Tri, Hue)

- Data used: Data was collected during project implementation from 2008 up tonow

- IS referred in the thesis is in formal, organizational computer-based informationsystem setting on accepted and fixed definitions of data and procedures forcollecting, storing, processing, disseminating and using the data

- The purpose of Information System is to serve and support M&E in project notfor general management

CHAPTER 1: THEORETICAL BACKGROUND ON INFORMATION SYSTEM (IS) AND INFORMATION SYSTEM DEVELOPMENT

METHODS

1.1 Theoretical background on Information System

1.1.1 Definition of Information System

Regarding to the IS, there are some concepts which need to be clarified toexhaustively understand the nature of research problems, those concepts areinformation and data

“Data are streams of raw facts representing events occurring in organizations or the

physical environment before they have been organized and arranged into a form thatpeople can understand and use” (1,p7)

Information are “data that have been shaped into a form that is meaningful anduseful to human being” (1,p7)

An Information system can be defined as a set of interrelated components thatcollect (or retrieve), process, store, and disseminate information to support decisionmaking, coordination, control, analysis and visualization in an organization (1,p7)

1.1.2 Information system perspectives

An information system represents an organizational and management solution,based on information technology to a challenge posed by the environment To fullyunderstand information systems, a manager must understand the broaderorganization, management and information technology dimensions of systems (TheMTO model) (see Figure 2) and their power to provide solutions challenges andproblems in the environment

by new knowledge and information system It is important to note that managerialroles and decisions vary at different levels of organization All level of managementare expected to be creative, to develop novel solutions to a broad range ofproblems Each level of management has different information needs andinformation requirements And information system used as an effectivemanagement tool to serve decision-making

Technology perspective

Information Technology is one of many tools managers use to cope with change.Computer hardware is physical equipment used for input, processing and outputactivities in an information system It consists of the following: the computerprocessing unit; various inputs, output and storage devices; and physical media tolink these devices together Computer software consists of the detailed,preprogrammed instructions that control and coordinate the computer hardwarecomponents in an information system Storage technology includes both the

Organization

Information System

Technology

Management

physical media for storing data, such as magnetic or optical disk or tape, and thesoftware governing the organization of data on these physical media.

Communication technology, consisting of both physical devices and software, linksthe various pieces of hardware and transfer data from one physical location toanother Computers and communications equipment can be connected in networksfor sharing data or resources Powerful computers, software and networks, includingthe Internet, have helped organizations become more flexible, eliminate layers ofmanagement, separate work from location, and restructure workflows, giving newpowers to both line workers and management Information technology providesmanagers with tools for precise planning, forecasting and monitoring of operation

To maximize the advantages if information technology, there is a much greater need

to plan the organization’s information architecture and information technologyinfrastructure

Organization perspective

The key elements of an organization are its people, structure, operating procedures,politics and culture An organization coordinates works through a structuredhierarchy and formal, standard operating procedures (SOPs) which has beendeveloped for accomplishing tasks, guiding employees in a variety of procedures.Organization operation is based on these SOPs and they are corporate intoinformation system Each organization has a unique culture, values and ways ofdoing things They are sometime found embedded in its information system.Different levels and specialties in an organization create different interests andpoints of view And also managers can not successfully design new systems orunderstand existing systems without understanding organizations

1.1.3 Functions of IS in organization

According to Kenneth C Laudon and Jane P.Laudon, 2005, an information systemcontains information about an organization and its surrounding environment Threebasic activities – in an information system produce the information thatorganizations need to make decisions, control operation, and analyze problems.These activities are Input, processing, and output (the IPO model) (see Figure 3)

Source: (1, P7)

Figure 3 The IPO model - Functions of IS in organization Input is “the capture or collection of raw data from within organization or from its

external environment for processing in an information system”

Processing converts the raw input into a more meaningful form to human This

meaningful information will be stored in standard formats and they are sometimethe base for making decision of the organizations

Output transfers the processed information to the people who will use it or to the

activities for which it will be used

Feedback is output that is turned to the appropriate members of the organization to

help them evaluate or correct input

1.1.4 Impacts of information systems on organizations

Information systems and the organizations in which they are used interact with andinfluence each other The introduction of a new information system will affectorganizational structure, goals, work design, values, and competition betweeninterest groups, decision making and day-to-day behavior At the same time,information systems must be designed to serve the needs of importantorganizational groups and will be shaped by the organizational structure, task,

Feedback

goals, culture, politics and management Information technology can reducetransactions and agency costs, and such changes have been accentuated inorganization using the internet The information systems departments are the formalorganizational units that are responsible for the organization’s information systemsfunction Organizational characteristics and managerial decisions determine the rolethis group will actually play.

Table 1 Positive and negative impacts of information systems

Benefits of information systems Negative impacts

Information systems can perform

calculations or process paperwork much

faster than people

By automating activities that werepreviously performed by people, informationsystems may eliminate jobs

Information systems provide new

efficiencies through services such as

telephone system or computer-controlled

process

Information Systems are used in so manyaspects of everyday life that systems outagescan cause shutdown of operations,paralyzing communities

The internet distributes information

instantly to millions of people across the

world

The Internet can be used to distribute illegalcopies of software, books, articles and otherintellectual property

Source: (1, P31)

1.2 Information system development methods

According to Laudon, systems differ in term of their size and technologicalcomplexity and in term of the organizational problems they are meant to solve.Hence, a number of methods have been developed to build systems: the traditionalsystem lifecycle; prototyping; Application Software packages; Internet connection;End-user development; Outsourcing

System Development Life Cycle (SDLC) is a traditional methodology for

developing an information system that partitions the system development processinto formal stages that must be completed sequentially with a very formal division

of labor between end users and information system specialists The system lifecycleapproach can be costly, time consuming and inflexible Although system builderscan go back and forth among stages in the lifecycle, the system lifecycle ispredominantly a “waterfall” approach in which tasks in one stage are completed

before work for the next stage begins Activities can be repeated but volumes ofnew documents must be generated and steps retraced if requirements andspecifications need to be revised This approach is also not suitable for many smalldesktop systems, which tend to be less structured and more individualized

Prototyping is the process of building an experimental system quickly and

inexpensively for demonstration and evaluation so that users can better determineinformation requirements The prototype is the preliminary working version of aninformation system for demonstration and evaluation purposes Prototyping is mostuseful when there is some uncertainty about requirements or design solutions.Prototyping is especially useful in designing an information system’s end-userinterface (the part of an information system through which the end users interactwith the system, such as outline screens and commands) However, rapidprototyping can gloss over essential steps in systems development If the completedprototype works reasonably well management may not see the need forreprogramming, redesign, or full documentation and testing to build a polishedproduction system Some of these hastily constructed systems may not easilyaccommodate large quantities of data or large number of users in a productionenvironment

Application software package

Information system can be build using software from application software packages

If the software can fulfill most of an organization’s requirements, the company doesnot have to write its own software The company can save time and money by usingthe prewritten, pretested software programs from the packages If an organizationhas unique requirements that the package does not address, many software includecapabilities for customization When a system is developed using an applicationsoftware package, the organization will issue a Request for proposal (RFP) which is

a detailed list of questions submitted to vendor of software or other services todetermine how well the vendor’s product will meet the organization’s specificrequirements This method can save time and money but in case of great deal ofcustomization, this may become so expensive and time consuming And when the

software selected, the organization no longer has total control over the systemdesign process Instead of tailoring user requirements, the design efforts will consist

of trying to mold user requirements to conform the features of the package

Outsourcing

Outsourcing is the process of turning over an organization’s computer centeroperations, telecommunication networks or applications development to externalvendors There are two forms of outsourcing, an organization can hire an externalvendor to design and create the software for its system, but the company can operatethe system on its own computer or left the operation to the vendor also

Outsourcing has become popular because some organizations perceive it asproviding more value than an in-house computer center or information system staff.Outsourcing allow an organization with fluctuating needs for computer processing

to pay for only what it uses rather than to build its own computer center, whichwould be underutilized when there is no peak load And the disadvantage ofoutsourcing can create serious problems for organization if they are not wellunderstood and managed Many firm underestimate cost for identifying andevaluating vendors of information technology services, for transitioning to a newvendor and for monitoring vendors to make sure they are fulfilling the contractualobligations These “hidden cost” can easily undercut anticipated benefits fromoutsourcing When a firm allocates the responsibility for developing and operatingits information system to another organization, it can lose control over itsinformation systems function If the organization lacks the expertise to negotiate asound contract, the firm’s dependency on the vendor could result in high costs orloss of control over technological direction An organization is most likely tobenefit from outsourcing if it understands exactly how the outsourcing vendor willprovide value and can manage the vendor relationship

Table 2 Comparison of Systems-Development Methods

Method Features Advantages Disadvantage

System

lifecycle - Sequential

step-by-step formal process

- Massivepaperwork tomanage

participation

- Inappropriate forlarge, complexsystems

- Can gloss oversteps in analysis,documentation andtesting

- Save time, money whendeveloping common businessapplications

- Reduce need for internalinformation system resources

- May not meetorganization’s

unique requirements

- May not performmany businessfunctions well

customization raisesdevelopment costs.Outsourcin

g

- Systems built and

sometimes operated

by external vendor

- Can reduce and control cost

- Can produce systems wheninternal resources are notavailable to technicallydeficient

- Loss of controlover the informationsystem functions

Source: (1, p470)

1.3 Criteria to evaluate an Information System

A lot of internal and external factors influencing on the IS design and use In fact, it

is difficult to exactly measure the successful level of an IS However, some criteriawith scientific basics for evaluating the success level also mentioned by Laudonwhich can be used for evaluation:

(i) High levels of information system use: this criterion is mainly measured by

the number of system users

(ii) Users’ satisfaction with the information system: the satisfaction must be

derived from both internal and external users Because an organization is always in

a close relationship with various stakeholders The need for information searching,sharing and exchanging is very high And the users often concern about thetimeliness, content and format of processed information These are also three sub-criteria for measuring if an information system is successful or not In manyresearches on Information system, these criteria are often mentioned as main criteriafor assessment on an Information system N.Ahituv and S.Neumann, 1979, alsomention about these criteria The user satisfaction mainly depends on ISeffectiveness in term of timeliness, contents and formats:

Timeliness is a class of attributes all related to the time factor in information update

and retrieval In term of timeliness, there are two concepts mentioned those are

currency and frequency Currency is the time for recording, processing the date and

responding from the user When this time is very short we describe the system as sreal-time system Real-time system is feasible only if an online technology isadopted Frequency is about the timing and cyclicality of information which help usupdate the data and support decision making

Content attributes relate to the meaning of information to decision makers For top

or middle level management, information content is more crucial than timeliness.Hereafter are some content’s attributes:

+ Accuracy: Importance level of information varies with types of decisions.

Long-range strategic decision normally need less accurate than low – level promptdecisions Accuracy can be measured by comparing actual data to the true eventstheoretically but it is not always applicable practically because we do not oftenknow “true” events and we prefer ex ante assessment in spite of ex post evaluation.Overcoming this problem is by either taking sample of data and performingstatistical estimates or investigating the quality of the keying and editing functions

in information system More accurate data can be expected whenever a morecomprehensive validation check is involved.

+ Relevance: Information is relevant if it leads to a different decision It might

also be relevant it reaffirm a previous decision If It does not have anything to dowith your problem, it is irrelevant In practice, we try to design relevant reports afterperforming a qualitative information requirement analysis (Munro, 1978) andimposing the requirements on the reporting design

+ Exhaustiveness: This feature goes further relevance It can fulfill one’s need of

information for better making decision Information requirements analysis may help

to measure this feature although it is not the ultimate way

+ Redundancy: Generally redundancy is not bad when we shed or stress some

data but it is boring and may cause negative reaction if we exaggerate The possiblemerits of redundancy can be measured by experiments

+ Level of details: Too detailed data might confuse, mislead and prevent the user

from acquiring comprehensive knowledge of a problem area It is likely that foreach decision problem there is an optimal level of detail (not too much and not toolittle) Theoretically the optimal level of detail is defined as the level under whichany more detailed information would not improve our decision and above whichany further aggregation would worsen the decision In practice it is difficult toqualitative analysis of aggregation requirements

Format of reported information can be evaluated through the possible attributes such as:

+ Medium: by which the report is provided, such as printed, visual display,

plotter (graphs), microfilm ECT

+ Ordering: the way the data are arranged in the report, such as the order of

the columns (horizontal ordering), the sequencing of details and totals (verticalordering)…

+ Graphic design: the graphic setting of the report, such as colors, letter sets,

and font…

All above format attributes must satisfy the criteria of simple and friendlyformat

(iii) Favorable attitudes of users toward the information system: In case of

high satisfaction of users, it often results in user’s favorable attitudes However, formany reasons, even high satisfaction of user they also do not show their favorableattitudes In that context, the Information system is not considered successful

(iv) Achieved objectives of information system: Each IS designed to serve a

certain managerial purpose and has different scope, some IS to serve the overallmanagement of an organization as ERP, but also there are some IS serving aspecific professional department or sections such as accounting software, personnelmanagement software…Thus, IS success level can be measured in term of purposeachieved (scope of IS, information shared, the number of users…)

(v) Financial payoff in application of the information system: It means that

the IS should be a cost – effective based system and in accordance with eachorganization’s resources Also the IS brinks economic benefits such as time-saving,reducing transaction costs and other intermediate costs in IS performance This isone of the important criteria which should be much paid attention for IS successmeasurement

CHAPTER 2: MONITORING AND EVALUATION (M&E) AND INFORMATION SYSTEM (IS) SUPPORTING M&E IN ODA PROJECTS

2.1 Overview of Monitoring & Evaluation

2.1.1 Definitions of M&E

Monitoring

As indicated in the Decree No 131/2006/ND-CP, monitoring is regularly andperiodically updating almost information related to project implementation,classifying and analyzing information, and timely proposing solutions servingmanager’s decision-making in order to ensure that the project’s objectives achieved

in scheduled and qualified under the defined resources Monitoring firstly is aninternal activity – an important part for good management, so it is an integral part indaily works Monitoring is the responsibility of the Executing Agencies (PMUs, theGoverning Body) and sometime supported by contractors or consultants

Main duties of monitoring in project implementation phase:

- Ensuring to adopt information demand serving management

- Coordinating information collection and analysis

- Support to collect and exchange unofficial information

- Support to hold the periodic review meetings among all implementing units

- Preparation to supervising delegations

- Announcing results to stakeholders

- Preparing quarterly and yearly reports according to requirements

Evaluation

As indicated in the Decree No 131/2006/ND-CP, evaluation is defined as a regularaction to comprehensively and objectively review the relevance, effectiveness andefficiency of the project in order to have timely adjustments and lesson learns to thefuture works

Evaluation is implemented in 4 phases of investment process: Inception

evaluation; Midterm evaluation; Completion evaluation; Impact evaluation or afterproject evaluation

Distinguishing between Monitoring and Evaluation

Monitoring and Evaluation are clearly different functions in investmentprocess and they serve various users Differences between Monitoring andEvaluation are summarized in the following table

Table 3 Differences between Monitoring and Evaluation

Continuous or periodic Periodic or sudden

Objectives are implemented as proposed Program objectives are evaluated with higher

objectives or developing issues which needsolving.,

Identified progress indicators are

considered appropriate

It is allowed to question the effectual level andappropriateness of identified progressindicators

Progress monitoring by a few previously

identified indicators

Solving various issues under different scopes

Focusing on expected results Expressing expected and unexpected resultsQualitative methods Both qualitative and quantitative methods

Daily collected information Several material sources

No answers for cause and effect

questions

Yes

Usually an internal management function It is often done by external evaluation

specialists and proposed by independentagencies

Source: ODA project management (2007), VAMESP II

The relationship between Monitoring and Evaluation

Although Monitoring and Evaluation are very different but closely related to eachother, they support each other and play equal important roles Monitoring providesquantitative and qualitative data by the use of selected indicators and thisinformation becomes input for evaluation

Basically, outputs of monitoring are direct inputs of evaluation But evaluation alsoprovides the support and lessons learned to Monitoring The following figure showsthe operation area of Monitoring and Evaluation This figure helps to illustrate theinterface of Monitoring and Evaluation.

Inputs Activities Outputs Results/

Figure 4 “Operation area” of Monitoring and Evaluation

Monitoring and Evaluation are clearly different functions in investment cycle.However, in practice, Monitoring and Evaluation is a continuous process frommonitoring to evaluation until the investment cycle is completed and the target

tower of logical frame starts from inputs to final goals/objectives Hereunder

indicates the M&E data process which shows the relationship between Monitoringand Evaluation

Source: Textbook of ODA project management – M&E module, 2007

Figure 5 M&E data process

Notes:

Continue toUnderThe above figure shows that Monitoring and Evaluation are different but alwayscombined in implementation process to reach the final outcome reports

2.1.2 Monitoring & Evaluation classification

ODA project M&E are classified according various factors as follows

M&E implementation

Data recording

and aggreagting

Outcome reports

Requirements on baseline data

Aggregated in database or worksheet

Table 4 M&E classification

Monitoring types - Internal monitoring

- External monitoring

- Internal evaluation

- External evaluationMonitored

objects - Disbursement (Finance, investment)- Implementation (Resources,

Source: Author’s analysis

In general, porject implementation monitoring is the internal monitoring performed

by porject parties Evaluation is often done well if the method of external evaluationbecause the external M&E agencies or consultants ofren have much experience

2.1.3 Role of M&E in project implementation

M&E are two best useful tools which help ODA managers in (i) making plan and

exposing essential adjustments (ii) ensuring project expected progress, results andimpacts Each ODA project is aimed at one or several eco-social objectives andthese objectives are identified right from the project identification phase M&E arevery important to consider if project can achieve these objectives effectively or notand recommend required conditions to ensure that the objectives will be obtained Incase M&E system is weak or M&E activities are not strictly implemented, it willcause ODA managers lack of information of three basic contents, those are (i)whether project has been implemented as planned or not; (ii) if project brings results

as designed or not; (iii) and how are the impacts of project M&E are to supportmanagement towards development results With the increase of investmentmethods, ODA performance also needs proving its effectiveness in order to meet allrequirements of participating units Management towards development results is agood method matching these requirements Management requires recording,monitoring inputs, outputs and outcomes in order to measure the success ofdevelopment results

2.1.4 General regulations on ODA project M&E

ODA project is the official development assistance project which is under thedevelopment activities framework of the Government and sponsors So, ODAprojects must follow both the requirements of the Government and Sponsors inproject implementation Generally, sponsors (ADB, WB, IFAD act) will have theirown requirements for M&E These are considered rather perfect systems.Meanwhile, the Government of View Nam has already put forth many regulations

on M&E for ODA projects through a lot of legal documents such as Decree 131,Decision 803 ECT So, the most importance thing that project must build an M&Esystem which can combine both sides’ requirements

As stipulated in Decree No 131/2006/NĐ-CP, M&E responsibilities are as follows:

- PMUs make the overall and detailed Workplan and in which the resources, theimplementation progress, the completion time, the quality target and criteria foraccepted results to each activity are clearly defined to be the base for M&E Theplans must be built based on the agreement with the Donor and submitted to the

EA for approval

- Setting up the internal information system, adequately collecting and storing

information, data, files, documents, and contractor’s reports, changes in the Government policies and legislation and the Sponsor’s regulations related to project implementation

- Making reports under requirements, providing and sharing information via thesector, local and national level of M&E

- Hiring consultants inception, midterm and completion evaluation

Through the above regulations, setting up an information system is a duty that anyODA project must follow to ensure the effectiveness of M&E

2.1.5 M&E steps

Monitoring and Evaluation are in close linking but in order to build up an M&E

system, each parts must be build in separate steps as indicated hereafter.

Monitoring steps:

An effective monitoring system is built and implemented in nine following steps:

- Step 1: Logical frame development and adjustment

- Step 2: Identifying activity questions and indicators

- Step 3: Making monitoring frame and plan

- Step 4: Organizational preparation for qualified monitoring

- Step 5: Preparation of equipments and IS

- Step 6: Data collection and analysis

- Step 7:Reporting project implementation results and monitoring information:

- Step 8: Assisting on the use of M&E information and feedback

- Step 9: Review essential conditions and capacities

Evaluation steps:

Evaluation can be carried out through the following steps:

- Step 1: Making evaluation logical frame

- Step 2: Making evaluation plan

- Step 3: Data collection and analysis

- Step 4 Evaluation conclusion

One of important requirements in M&E building is that it is essential to harmonizethe Gov and the Donor’s regulations and other regulations stipulated in theLoan/Grant Aggrement Therefore, in case the Donor has already set up its ownM&E system regulations, the project must follow that system in comparision withthe Gov’s one to set up the most relevant system

2.2 The supportive relationship between IS and M&E

Through the approach and analysis of M&E system, it is more clearly demonstratedthat information system is to serve M&E purpose Information management is aprocess in which the project information is created, used and shared, and this is one

of important phase in M&E performance, contributing to ensure the timely and

effective decision making The following figure is the model of supportiverelationship between IS and M&E

Source: Textbook of ODA project management – M&E module, 2007

Figure 6 The model of supportive relationship between IS and M&E

Notes:

IncludeContinueSupportive relationshipThe figure shows that IS and M&E are always in closely supportive relationship.The M&E frame with relevant indicators will stipulate what information need to bemanaged, who and how to collect data, how to measure project outcome ECT IS is

a useful tool to systematically manage the information serving M&E purposes:Implementation, Compliance and Impact M&E Once the IS is well functioned, thisnot only contribute to effective M&E but also to smooth information flow to serveproject implementation and management

2.3 Principles to build an information system supporting Monitoring & Evaluation

Logical frameM&E frameReport’ requirements…

M&E system

Information system

Sponsor’s requirement)

M&E plans with

indicators reported

The IS building to support M&E must ensure the following principles:

- IS staff must be trained on the process of information creating, using andsharing

- IS staff must take the initiative in information management

- The IS must be built since the project inception focusing on: (i) therequirements on project information management (to serve M&E for example)(ii) staffing to information management, (iii) scope of managed information,(iv) deciding the method of information management (software, worksheet,management information system …)

- Various support management tools must be invested to improve the effectiveinformation management: computers, projector, storage devices, internet,internal networks…

- There must be the participation of stakeholders in IS performance

- The managed information must be processed, exact, unified and officialinformation

2.4 Experience in applications of Information system supporting M&E in some ODA projects in Viet Nam

Basing on the general regulations and separate requirements of Donors as well asdifferent features on management, organization and different working conditions,each project has their own M&E with specific characteristics Accordingly, eachproject has different decision on selecting the IS to the project

In this section, the author wants to present the lesson learned and experience inapplication of IS supporting M&E in other ODA project in term of IS functions andperspectives Two typical examples reflected in the thesis are the IS applied in RoadNetwork Improvement Project (RNIP) funded by the World Bank and Viet NamLand Administration Project (VLAP) also funded by the World Bank

2.4.1 Experience in application of Information system supporting M&E in

Road Network Improvement Project (RNIP) funded by the World Bank

Road Network Improvement Project (RNIP) funded by the World Bank has beencarried out in 14 provinces Through combining the method of observation and in-depth interview to a vice director, it was expressed that the IS applied to the wholeproject serving M&E system is the Aligned Monitoring Tool (AMT) built andintroduced by the Ministry of Investment and Planning (MPI)

AMT is an IS with good graphic design and full functions of Input – Processing –Output As an information/data is inputted, the related sheets are automaticallyprocessed and changed At the home page of AMT, there is a list of linked tables

classified in groups of data (See Appendix 5) Indepth interview with Mr Linh

-deputy director of RNIP showed that the project took advantage of AMT for storingthe essential information of the entire project, especially the disbursement progress,procurement details, activities completion level ect Basically, AMT was able tomeet with key contents for M&E But comprehensively, AMT is unable to cover allneeded information for three main purposes of M&E due to some problems asfollows:

- AMT only can cover all monitoring contents because almost data input is

quantitative data

- AMT updating and reporting mechanism faced with a lot of restrictions: InAMT, the comparison of plan and actual data is combined in the same sheet,the formats were unable to show the detailed data of almost plans in differentaspects which need to be monitored and evaluated For example, the systemdid not have the separate sheets of overall and detailed implementation plans,ethnic frame, personnel plan, training plan and other management plans ect.AMT also lacks of an important part is that the overall M&E plan which wasthe base for the entire M&E IS and for making plans

Regarding to organization, management and perspective, he also said that "the current AMT has been considered as a management tool supporting the management levels on decision making" M&E staff were already trained on data

collection, AMT operation and reporting by MPI’s training course However, thesystem has not really met with the requirements in term of organization and

management aspects under M&E mission Reporting mechanism and informationsharing capacity are the most visible manifestation of limitations underorganization and management perspectives AMT has been applied to all levels inthe project Each province has separate AMT On quarter basic, the provincescomplete their system with manager’s approval, and then the system is submitted

to the central level for aggregation via email The central staff aggregate theinformation based on collected AMT files Aggregation method used is a manualmethod; the data can not be integrated into a system Thus a lot of problemsaraised:

- The information aggregation caused a lot of time and the information accuracy

is not high

- AMT is also unable to ensure good information sharing due to an offlinesystem The information is not available for stakeholders' concerns Therelated agencies (Project Steering Committee, suppliers, contractors ect) canget information only when the information posted into the Website or M&Ereports submitted as requested

- In addition, in the same system AMT is also unable to satisfy the differentinformation demands of various management levels and related agencies Theprovincial and grass-foot levels need to have as detailed information aseffective while the central levels need the aggregated information

In terms of technology perspective, the project has already equipped with physicaldevices such as computers, storage devices and the external and internal network(Internet, LAN within a PMU) ECT However, there is no proper IS which canautomatically store and process the information effectively via online method.AMT is simple software which is unable to cover almost main requirements ofboth Dornor and the Gov on M&E

2.4.2 Experience in application of Information system supporting M&E in

Viet Nam Land Administration Project (VLAP) funded by the World Bank

Viet Nam Land Administration Project is funded by the World Bank and governed

by the Ministry of Resource and Environment VLAP has been carried out in 09provinces

Currently, the IS supporting M&E in VLAP has been completed and in operation

By observation and in-depth interview, the findings showed that the IS quite properand well-functioned with input-processing-output This IS basically satisfies therequirements at organizational, management and technical aspects

Mr Mau - Project coordinator said that “to build an IS supporting M&E the project must catch up with the regulations of the GoV and the Donor on M&E and other requirements in term of organization, management and technology perspective to the IS supporting M&E” Those regulations clearly indicated in the Project

Administration Manual, Decree No 131/2006/ND-CP by the Government on themanagement and use mechanism of ODA fund, Decision No 803/2007/QD-BKH

by the Ministry of Investment and Planning on the mechanism of periodic reporting

in ODA project ect He also said “We all know that it is important to build an IS at the first time of project implementation Thus, the leading board in the Central Project Management Unit (CPMU) organized a meeting to review the project resources for building an IS supporting M&E” Basing on these requirements, a

Request For Proposal (RFP) was prepared for consulting service of designing andinstalling the IS software supporting M&E The software was called

“VLAP_M&E_IS” The successful vendor proposed a relevant solution which is inoperation now and initially brink back a lot of benefits

The IS solution given by the vendor clearly showed the real state of IS applied in

VLAP in term of functions and perspectives See Appendix 6 - Summary of

VLAP_M&E_IS solution for detailed information

Through the analysis of VLAP_M&E_IS solution, some important findings are summarized as follows:

- VLAP_M&E_IS is a system which simple, friendly and unified formats (hastaken the advantage of AMT formats to the IS functions as many as possible)and relevant indicators; The IS automatically processes information via onlinemethod; Timely reports and other materials produced; Available information forstakeholders' concerns

- The IS used as a management tool with user’s decentralization; Access dairy;Available information of all levels by the time which serving decision- making

- IS and M&E staff are all methodically trained and they have enough capacity toundertake their tasks; clear cut responsibilities in IS performance

- The IS is a Web-based software with advance programming language

VLAp_M&E_IS is quite a proper and reasonable system applied to ODA projectwhich has a lot of components and carried out in many provinces The systemensures the close connection between the central levels and the lower implementingunits with smooth information flow However, the system also has a limit oftechnical errors happening which exceed the self-treatment of PMU staff Thus insome cases, the repairment cause delay and affect on the common informationprogress

The findings on IS application in RNIP and VLAP help us get some lessons learnedand experience as follows:

Based on IPO model - IS functions: Well functioned IS with

input-processing-output; Both qualitative and quantitative data reflected; Good graphic design,simple, clear and friendly formats; Automatically processing and aggreatinginformation to provide available information

Based on MTO model - IS perspectives:

- Management perspective: The system must be used as a management tool to

serve the purpose of M&E: IMPLEMENTATION, COMPLIANCE ANDIMPACTS M&E

- Technology perspective: It is essential to build a relevant IS which is able to

combine the physical devices, the software and internet into a proper softwarewith online method This system will supply a constantly updated information,automatically processed information and integrating all information of theentire project Other technology investments also should be prepared to serve

IS operation

- Organization perspective: Full-time M&E staff and they must be carefully

trained on M&E as well as IS M&E frame must be properly built withrelevant indicators in accordance with project objective and outcomes to bethe base for making plan and definition of managed information To operatethe IS well the project must create a well-organized system with clear cutfunctions and responsibilities of all levels and staff involved in the IS process.Those above lesson learned are very useful to identify the requirements to theimproved IS in STDPVN Although VLAP and STDPVN are different in thesource of fund but they have the same regulations on M&E by the Gov, the IS oftwo project are only different in formats and indicators stipulated in M&E system

of each project In case this IS model of VLAP applied in STDPVN, it is onlyneeded to be customized under specific STDPVN’s requirements One moreimportant thing noticed that to comprehensively analyze and proposeimprovements to an IS, the IS is not only considered at its technical aspects butalso its management and organization perspectives