SIDP Evaluation Report final June 2014

ACRONYMS AND ABBREVIATIONSAGO Accountant General Office BCPR Bureau for Crisis Prevention and Recovery CoA Chart of Accounts CSC Civil Service Commission CSI Civil Service Institute DAD

REPORT FINAL EXTERNAL EVALUATION OF SOMALI INSTITUTIONAL DEVELOPMENT

PROJECT, UNDP, SOMALIA

P MADHAVA RAO MICHAEL PALMBACH KERRY ABBOTT

Evaluation of Somali Institution Building Project that was launched by UNDP under some noble compulsions is not an easy task A project that was formulated for Institution Building for a Nation with large budget estimate and HR component really needs to be evaluated with sincerity and commitment considering the circumstances and conditions under which it was formulated and launched

The evaluation team sincerely, thanks all those who have honestly supported the team to accomplish this otherwise tough task

The comments offered by Mr David Akopyan the deputy country director, Mr Albert Soer, the Chief Technical Advisor, Mr Franco Sanchez, the Rule of Law and Governance programme, Mrs Christine Fowler the acting project manager till February 2014, Mrs Bushra Hasan, the UNDP M&E Officer, , were of great importance and added value to the report But for their insights into the conditions under which the project formulated and implemented, the evaluation finding could have been incomplete The team members thank them from the core of their hearts.

The technical, secretarial, and logistical support provided Mr Sergei Pushkarov and Ms Samira Hassan, the SIDP project team at Hargeisa; hospitality and insight into project management provided by Mr Stephen Kinloch Pichat, the head of sub office at Hargeisa;

Mr Musa Duale, SIDP Project Officer at Garowe; Mr Mohamed Barre SIDP Senior Advisor at Mogadishu has been immensely helpful, and the evaluation team places on record its heartfelt appreciation of the support extended by them And in the home-office the assistance of Ms Keziah Kuria and Mehmood Hassan has greatly facilitated the work of the team The team owes them a lot.

The team thanks all those Ministers, Deputy Ministers, Vice Ministers, Director Generals, and senior functionaries in the line Ministries, Civil Service Commission, and Civil Service Institute, Offices of Auditor General, and the donors who have shared their candid opinions about the implementation of the project without any hesitation The report could have not gone to the root of the project implementation without their keen observation and understanding of the project results.

The information shared by the National Staff teams at Hargeisa, Garowe, and Mogadishu has been of great value Their perspective on future direction of the project has been deeply appreciated

Team takes pride in thanking the support extended by the transport team, travel unit, finance team, and security team of UNDP Somalia who made the living of the team members comfortable, trouble free, and memorable.

Finally, the evaluation team thanks those all who have been instrumental in giving a shape

to this evaluation, and requests the pardon of those whose names are not found in this acknowledgement Non-mention of the names here is a space constraint but not deliberate Thank you all.

P Madhava Rao Michael Palmbach Kerry Abbot

iii

ACKNOWLEDGEMENT ii

Contents iii

ACRONYMS AND ABBREVIATIONS v

EXECUTIVE SUMMARY vi

1.0 Introduction 9

1.1 Evaluation Themes and Evaluation Questions 9

1.1.1 Public Sector and Civil Service Reform 10

1.1.2 Public Financial Management 10

1.1.3 Development Planning and Aid Coordination 11

1.1.4 Evaluation objectives and combined questions: 11

1.2 Brief Description of the Project 12

1.3 Methodology of Evaluation 14

2.0 Findings 16

2.1 Review of the project document 16

2.2 Review of Implementation Arrangements 17

2.2.1 Project Successes 19

2.2.2 Project Shortfalls 22

3.0 Evaluation of three themes 24

3.1 Public Administration and Civil Service Reform- Overall findings 24

3.2 Public Financial Management 27

3.2.1 SIDP attempted to do too many things in PFM 28

3.2.2 SIDP results in PFM were only partially achieved 28

3.2.3. Public Financial Management - Findings specific to Somaliland, Puntland, and South-Central Somalia 29

3.3 Development Management and Aid coordination 34

3.3.1 Project Design 34

3.3.2 Project Analytical Framework 35

3.3.3 Implementation Review of SIDP 38

3.3.4 Development Planning and Aid Coordination Objectives 40

3.3.5 SIDP Assistance to Development Planning 41

3.3.6 SIDP Assistance to Aid Coordination 44

3.3.7 Challenges facing SIDP 47

3.3.8 Evaluative Development 48

3.3.9 Findings on Development Planning and Aid Coordination 49

4.0 Overall Relevance, Efficiency, Effectiveness, and Sustainability of the Project 51

4.1 Relevance of project 51

4.2 Effectiveness 51

4.3 Efficiency 52

4.4 Sustainability 53

5.0 Lessons learned 55

6.0 Moving Forward 58

6.1 Opportunities 58

6.2 Conclusions 59

6.3 Recommendations 62

Annex I - TOR 69

Annex 2 – Report and Documents 75

Annex 3 - List of Persons consulted and Field Visits 79

Annex 4 - Meeting Schedule for the Evaluation Mission 82

Annex 5 – Evaluation objectives and questions answered 87

v

ACRONYMS AND ABBREVIATIONS

AGO Accountant General Office

BCPR Bureau for Crisis Prevention and Recovery

CoA Chart of Accounts

CSC Civil Service Commission

CSI Civil Service Institute

DAD Development Assistance Database

DCD Deputy Country Director

ETA Emergency Technical Assistance Project

GFS Government Financial Statistics

GGC Somaliland Good Governance Commission

ICT Information and Communication Technology

IOM International Organisation for Migration

JLPG&SD Joint Programme of Local Governance and Service Delivery LoA Letter of Agreement

M&E Monitoring and Evaluation

MIDA Migration for Development in Africa

MoF Ministry of Finance

MoLSY Ministry of Labour, Social Affairs and Youth

MoPIC Ministry of Planning, and International Cooperation

MoPND Ministry of Planning and National Development

NCSC National Civil Service Commission

OAG Office of Auditor General

OPM Office of the Prime Minister

PAR Public Administration Reform

PFM Public Financial Management

PSU Puntland State University

QUEST Qualified Somali Technical Support

SCMSP Somali Constitutional Support Project

SICAD Somali Institutional Capacity Development Project

SIDA Swedish International Development Agency

SIDP Somali Institutional Development Project

SISP Somali Institutional Support

UNDAF United Nations Development Assistance Framework

UNDP United Nations Development Programme

USAID Unites States Agency for International Development

USD United States Dollar

EXECUTIVE SUMMARY

The Somali Institutional Development Project (SIDP) was an initiative byUNDP launched during 2009 and extended up to the end of 2013, whileduring 2014 still a number of smaller activities are finalised A numbercapacity development support projects operated by UNDP up to 2008 weremerged to give birth to SIDP The project initially proposed to mobilise andspend about USD 55M over a period of three years However, the projectcould not mobilise these funds and hence the overall scope of the actualimplementation was smaller than the initial intentions

In accordance with the provisions contained in the project document, SIDP ismandated to conduct an end of the project external evaluation, which isdone to provide value for future activities and to offer findings on the extent

to which the project met its objectives Accordingly, three internationalconsultants with expertise in public administration and civil service reforms,public financial management, and development planning and aideffectiveness were engaged by UNDP to conduct an evaluation However,apart from the formal requirement to evaluate the project, UNDP embarked

on this evaluation to document the achievements and challenges the projectfaced and to draw the lessons for the implementation of the future activities.The evaluators have taken 28 days from 25th April to 22 May to evaluate theproject and submit this report

The Somali Institutional Development project worked under great stress due

to security issues, changing governments, and inadequate staff and financialresources

Despite being understaffed, and underfunded, the project has been ableestablish itself as project which has provided support in a volatile securityand governance environment in Somalia The project significantlycontributed to making the civil service commission and civil serviceinstitution in Somaliland functional; the Ministry of Labour and Office of theAuditor General of Puntland are now working in a reformed environment Theproject has produced about 50 advisory, policy, and strategy notes andguidelines requiring follow-up There is a momentum towards overallinstitutional development in all the three regions after the launch of thisproject

vii

The project simultaneously suffered because of under-performance ofactivities taken up, non-recruitment of required staff in all 3 regions,insufficient monitoring of progress, and not enough follow-up with all thestakeholders Government ownership issues, sustainability of advice given,development of relations with all the stakeholders, ineffective reforms havebeen the areas of criticism.

Considering the conditions under which the project operated, the followingconclusions and recommendations are made

Conclusions:

 Partial achievement of goals: there was partial attainment of goals,

 Supply oriented product delivery: too many of the project

deliverables were supply oriented, and not tied into the existing

systems, i.e manuals were delivered which did not take into account the current status of the systems which were meant to be improved,

 Project ownership: the impression is wide-spread that reforms were

UNDP driven, and that the role of the government was to sign-on to the initiatives designed by UNDP consultant The created issues of

ownership,

 Staffing recruitment and retention: all required staff were not

deployed, retention was an issue,

 Capacity Development became secondary: focus was on capacity

substitution & equipment not capacity development (this also was partially due to the requirements met on the ground),

 Follow-up and in-depth technical assistance: deliverables were

often trainings and manuals, which introduced the topics but follow-on actions were often missing,

 Unclear vision and operational plans: project objectives, inputs,

outputs, and outcomes were not clearly stated,

 SIDP/UNDP response to issues: recognized short-comings, and took

steps to remedy them

Recommendations

 Project formulation, communications, refinements: Invest in

extensive communications, interaction, and continuous dialog to ensurethat the project is/and remains demand oriented,

 Donor coordination and communications: develop synergies with

other donors in their programs, establish and follow more formalized

communication mechanisms,

 Local alignment: align the project with Somali Compact, New Deal,

and the National Development Plans of the regions,

 Public administration reform alignment: stronger alignment with

public administration reforms (those planned and underway locally),

 Utilization of Somali expertise: cultivate, invest in, and utilize

Somali experts wherever possible,

 Sub-offices to service each State/Region: develop specific work

plans for South-Central, Somaliland, and Puntland, establish offices withsome delegated authority in each region,

 Intensive rather than extensive approach: selected limited

number/prioritised line ministries for deeper involvement, do fewer things better, learn to say “no”

 Do not undertake reform of PFM systems: do not undertake

systems-development in PFM (development of budget format,

standards, law writing, etc.), leave much of this for specialists,

 Cooperate with PFM initiatives: engage in PFM activities when they

interact with policy, governance, capacity development issues (i.e assisting ministries in planning and budgeting),

 Improve quality of technical assistants experts: develop a

pool/roster of institutional development experts (invest in the experts,

so that they may be called on repetitively),

 Develop strong management systems: develop strong

management systems and procedures, make the Project document the controlling instrument, amend it formally as required,

 Complete and/or follow-up SIDP activities: follow-up action on the

activities initiated by SIDP (continue to invest/complete initiative begun)where it serves new project objectives,

 Strong communication and monitoring systems: sub-offices,

clients, donors require both formalized and informal communications, this should be a two-way channel and utilized to influence the

implementation and directions of the project,

 Joint programming: joint programming with other actors with a rights

ix

1.0 Introduction

The Somali Institutional Development Project (SIDP) was signed by UNDP on

27th October 2008 with an estimated resource requirement of US $ 55 millionfor project duration of three years from 1st January 2009 to 31st December2011

The project document envisaged an end of project external evaluation at 36months, to evaluate the project against overall intended outcome However,

as the project has been extended twice, the end of project evaluation isbeing conducted now (May 2014) by three independent consultants Thefocus of the evaluation is mainly on public sector and civil servicedevelopment, public financial management systems, and developmentplanning and aid coordination These three areas encapsulate four pillars ofinstitutional development namely, Development of systems and processes ofmaking and implementing policy and law; Development of HumanResources; Improvement of public financial management (PFM); andDevelopment of physical infrastructure as promised in the project document.Following experts were contracted by UNDP for the purpose of evaluation:

 P Madhava Rao Team Leader and Public Sector and

Civil Service Reform expert

 Michael Palmbach Public Financial Management expert

 Kerry Abbott Development planning and aid coordinationexpert

The evaluators present this evaluation report after conducting availableliterature review, field interviews, and discussions with focus groups inSomaliland, Puntland, and South-Central Somalia A comprehensiveevaluation has been conducted by European Commission (EC) during 2012,and a comprehensive draft1 progress report was submitted by UNDP toEuropean Union on 10th March 2014 These reports present a good deal offacts about the project To avoid undue duplication, this evaluation reporthas limited itself to a few important issues and facts The report has beenimplemented according standards of evaluation enlisted in theUNDP/DAC/UNEG standards of evaluation, ethical issues, transparency, andfairness

1 At the time of implementing the present evaluation, the draft progress report was

submitted to the European Union However, the report had not yet been formally endorsed

or approved.

1.1 Evaluation Themes and Evaluation Questions

The present evaluation is an end-of-project evaluation Although someprojects really have a physical and precise closure date, in large andcomplex projects like SIDP minor activities often require continuation afterthe core of the project has closed

The SIDP project at the time of writing the present report was in the process

of finalising the remaining activities that had commenced earlier As per theAnnual Work Plan procedures of UNDP, SIDP is still an operational (althoughrather limited) project during 2014 financed through donor support fromSweden, Switzerland, Norway and through internal funds from BCPR and theTRAC funds The main donor (the European Union) to the project concludedits funding in September 2013, meaning that only those activities for whichfinancial commitments had been concluded before September could beimplemented after September with EU support With the submission of thedraft report in March 2014 to the EU, de facto, March 2014 can be withreason taken as the closure date of the core of the SIDP Hence, while still anumber of smaller and interim activities are ongoing; this report can beconsidered as the end-of-project evaluation

The Terms of Reference wanted the evaluation team conducts an end ofproject evaluation with specific focus on:

a Public Sector and Civil Service Reform,

b Public Financial Management, and

c Development Management and Aid Coordination

1.1.1 Public Sector and Civil Service Reform

Public Administration is the prime mover for service delivery, and civilservants make the public administration move Public Administration beginswith defining a clear mandate of the Government for public service delivery,and then it identifies functions of public service, structures, and then humanresource to fill up the structures Thus, it defines a policy and legalframework for public service delivery, sets up institutions, draws up workprocedures, guidelines, and processes, recruits, trains, and deploys civilservants Further, it places human resources management, and humanresources development law, policies, plans, and systems in place Thus, SIDPattaches prime importance to the public administration, and the peopleworking in it- the civil servants - and accordingly plans to institutionalisepublic administration and civil service management and development

11

1.1.2 Public Financial Management

Public Financial Management (PFM), while emphasizing the appropriation,use, and reconciliation of finances also brings into focus concepts ofmanagement, as well as the specific requirements necessary to work withinthe public sector Thus, PFM is a very broad topic, and does not belong only

to those who actually handle finance PFM in a developing country context,particularly where resources are scarce and where the technical skills toutilize funds may be missing, presents special development challenges

UNDP recognized the weaknesses inherent in Somalia, and devoted a portion

of the SIDP project to assisting in overcoming these conceptual and practicalshortfalls Broad support was provided to all of the main government units

to address; budgeting, accounting, treasury, and other PFM functions

1.1.3 Development Planning and Aid Coordination

Countries in conflict, post conflict, and fragile states have to drawdevelopment plans to effectively and efficiently use public funds In suchcountries, it is important the international aid is properly accounted for andthe Government coordinates with the aid agencies for effective relationsbuilding as well as efficient and effective deployment of the funds madeavailable UNDP Somalia thus gives due importance to the developmentplanning and aid coordination

The Terms of Reference for the consultants enlists the evaluation objectivesand evaluation questions These questions look deeper into four pillars ofinstitutional development envisaged in the project document, within therevised areas expertise being looked into Additional and relevant findingsand observations that add value to the purpose of evaluation as well areincluded into the evaluation

1.1.4 Evaluation objectives and combined questions:

 Question #1, Achievement of Intended Outputs: to examine to

what extent the project has achieved the intended outputs, and inwhat specific areas the project excelled or failed in progress towardintended outputs

 Question #2, Achievement of Intended Impacts/Outcomes: to

indicate whether or not intended project impacts and outcomes arebeing met and/or, for specific outcomes, whether satisfactory progresswas made

 Question #3, Factors Influencing Project Outputs and Impacts:

to analyse underlying factors that are influencing project impacts andoutputs

Implementation: to identify and analyse barriers and constraints that

have delayed implementation, including challenges emanating frominternal and external sources

 Question #5, Lessons Learned: To identify a list of ’lessons learned’

and make recommendations for corrections, including in relation to i)the specific CD services provided by the project (are they appropriatefor the client and do they respond to the demands and felt needs ofthe client?); and ii) the sourcing of capacity (what is the generalexperience and are steps be taken to source capacity from institutionsand networks beyond those considered under the project?)

 Question #6, Future Directions: Extract the lessons learned and

best practices that can be considered in the planning and design offuture support activities for government and recommendations forfuture direction and areas of focus for the next phase of project

 Question #7, Sustainability of Current and Planned Outputs: To

state whether or not targets are being achieved and whether currentand planned outputs can be sustained, including determination ofmeasures needed to ensure continued sustainability of results in thefuture

 Question #8, Issues of Security in Relation to Project Implementation: Review the project progress, recommendations and

action plan in light of the prevailing security situation in South andCentral Somalia, Somaliland and Puntland

The evaluation report considers all these questions However, answers maynot be in the same sequence or against each question separately

1.2 Brief Description of the Project

Capacity Development support to the Governments of Somaliland, Puntland,and South-Central Somalia to enable them to put in place viable and vibrantdemocratic institutions for public service delivery has been an importantendeavour of United Nations Development Programme (UNDP), Somalia.Accordingly, in pursuance of mandates of development needs of Somaligovernment, UNDP has launched its support initiatives for institution buildingthrough Somali Institution Building Project (SIDP), UNDP Somalia This project

13

has been launched after merging other smaller projects of UNDP like SomaliInstitutional Capacity Development Project (SICAD), Qualified SomaliTechnical Support (QUEST), Emergency Technical Assistance Project (ETA),and Somali Institutional Support (SISP) Additionally, other projects like JointProgramme of Local Governance and Service Delivery (JLPG&SD), and theSomali Constitutional Support Project (SCMSP) were expected to developsynergies with SIDP.

The Somali Institutional Development Project (SIDP) was signed by UNDP on

27th October 2008 with an estimated resource requirement of US $ 55 millionfor a project duration of three years from 1st January 2009 to 31st December

2011 The estimated balances of funds, to the extent of US $ 5,126,873 as

on 15th December 2008 were carried over from Somalia Institutional SupportProject (SISP) These came from Start Up Package Project (SUP), SomaliaInstitutional Capacity Development Project (SICAD), Emergency TechnicalAssistance Project (ETA), and Qualified Somali Technical Support Project(QUEST) to SIDP After appropriation of these funds against the requirement

of US $ 55 million, there were unfunded budget estimates of US $49,873,127.2 The available funds as on 27th October 2008 were:

2015, however, was not officially approved”

As of 28th December 2011, the total allocation of resources was US $ 29,731,930 funded by:

2 SIDP project document page 3

3 Amendment to Project Document signed by UNDP, 28 December 2011

Another extension six months no-cost retroactive extension was given up to

10th September 2013 on 27th June 2013 This extension starts from 11th March

2013 However, no document was made available with respect to anyextension between 1st July 2012 and 26th June 2013

1. Strengthening policy and legislative systems and processes;

2. Development of Human Resources’ capacities

3. Improving public administration through support to public financial management system (PFM) and public accountability,

4. Developing physical infrastructure and operational support

To accomplish these tasks the project document (2008) proposes amanagement structure with a project manager as the team leader, withsupport technical and operational staff at Nairobi, and three area projectmanagers to be positioned at Somaliland, Puntland, and South -CentralSomalia, with their technical and operational staff The project was launchedunder the overall mandate of Governance and Rule of Law programme ofUNDP Somalia

The project adopted an operational approach of delivering through Letters ofAgreement between partner Government institutions and SIDP where a clear

4 Project Document-Somali Institutional Development Project Dt.27/10/2008

15

distinction is made between the inputs to be delivered and managed by theUNDP and the partner, and where the different responsibilities in resultrealisation and project management are described.

1.3 Methodology of Evaluation

 Establish relevance, efficiency, effectiveness and sustainability

through review of documents, interview of stakeholders, and conduct

of focus group meetings from the civil service commissions in threeplaces, groups of interns, and Diaspora personnel deployed Theproject has produced a large quantity of documents, in the sense of (a)internal and external progress report and other moredonor/international partner related documents, in the sense of (b)consultant deliverables, and (c) documents produced by thegovernments where the project substantially contributed through theadvisors it deployed The sheer quantity of the documentation makes itvirtually impossible to review all and assess individually the possiblebenefit it may have had Instead, the mission adopted an approach togain insight in the collective contribution the different individual

‘outputs’ of the project may or may not have had in an overall changeprocess or reform In fact, we tried to develop the story of impact

 Translate findings with a forward-looking perspective into recommendations for ongoing and future work of the UNDP.

This approach is adopted mainly because there are already a fewextensive review documents and evaluation reports available Theevaluation team focuses more on forward looking perspective than onwhat has not been done However, lessons learned will be used as aguide for future planning

The evaluation report is drafted in simple, jargon free, and easy to readEnglish Subjective and ambiguous comments and statements like ‘goodprogresses; ‘bad management’; ‘excellent results’ etc are avoided as far aspossible and narration is based on measureable results and data as far aspossible The report follows the standard format of evaluation reports

Evaluation questions are answered for each of the thematic area ofevaluation, better appreciation of facts Each expert area is dealt with inseparate sections in the report to have focused appreciation of facts andfindings

The evaluation team was deployed for a total of 28 days including five dayseach in three regions The meeting schedule was developed with the hiringmanager, while additional meetings and discussions were conducted asrequired A presentation was made to country office team and donors on 20th

of May, and a draft final report submitted on 21st May

17

The evaluators place on record these challenges before proceeding furtherand before making any comments on performance or non-performance oftheproject.

2.1 Review of the project document.

Conceptually, the project was a well thought after initiative to integratecapacity development support extended by other projects of UNDP Theproject has realised the importance of core institutional development areassuch as: Policy and Legal framework; Human Resources Development, PublicAdministration reforms and Public Financial Management, and Infrastructure.The project has also identified critical success factors like distinctrequirement of different approaches of support to South-Central Somalia,Puntland, and Somaliland The project also has taken on board the load ofunfinished tasks under the past projects, for example, unfinished tasks underStart up Project (SUP) The project document also elaborates theconsultations with stakeholders before launch of SIDP

However, on the first reading of the project document, it appears that theproject document was written in urgency and insufficient consideration wasgiven to the results framework and UNDP standard structure of results basedmanagement Therefore, inconsistencies occur in the design specifically withreference to four pillars it had identified to support Public AdministrationReforms through support to public financial management was not explainedclearly in the documents and the activities under the pillar of public financialmanagement do not match with the requirement of public administrationreform In addition, the project document does not clearly speak out itsobjectives, and goals A clear resource mobilisation strategy for a USD 55Mproject could have been spelt out in the project document While, the projectdocument appreciated different approaches to different project locations,this was insufficiently translated into region-specific action plans as isevident from the results framework Nevertheless, the project has developedyearly work plans in later years of its implementation, and adopted anapproach of delivering through Letters of Agreement signed between theimplementing partners and UNDP, which allows for a very targeted andpartner-specific approach and action plan Lastly, the project document doesnot have the signature of implementing partner in token of their acceptance

of support from UNDP

2.2 Review of Implementation Arrangements

The implementation arrangements for SIDP as envisaged in the projectdocument have been threefold:

a SIDP under the overall supervision of a project management board, and a project headed by an international project manager; The project

was headed by an international project manager and his/her supportstaff to extend technical and operational support

b Implementation transferred to Government through a Letter of Agreement under the supervision of project staff; Letters of

Agreements have been entered into between government and UNDP

c Establishment of regional management units in Mogadishu, Garowe, and Hargeisa 5 Regional offices have been opened, but their staffinghas been a challenge for the project management and UNDP

5 At the time of writing this report (May 2014), the UNDP was in the process of

re-establishing its headquarters in Mogadishu With the exception of the period from January to June 2012, the Mogadishu presence is interpreted here as a ‘regional management unit’.

19

The organigram as presented in the original project document suits theproposed support to the government However, the proposed managementstructure was never implemented as designed The main reason for this isthat the project was not funded to the level as foreseen and the scope of theproject was subsequently less than foreseen Establishment of the originalmanagement structure would have been an over-investment and would haveled to relatively high project management costs As such, it is appropriatethat the originally foreseen structure was not implemented However, at thesame time – even with a less ambitious scope - the project did receivesubstantial funds and did embark on a rather diverse set of activities with avariety of partners, and as a result, the project remained complex andrequired a strong managerial support structure If the project managementstructures were established as planned, the project might have avoidedcomplaints of communication gaps, supervision gaps in project delivery,remote management, and delay in payment of promised money to thecounterparts under LoAs These are difficult choices, but with hindsight, it ishighly likely that the project would have benefited from a more robustmanagement structure.

2.2.1 Project Successes

SIDP has been able to establish its identity in the select line Ministriesirrespective of the security challenges, changing governments, intermittentproject management, and delays in the release of promised payments to theinstitutions under LoA Governments at all the three places - particularly thecivil service commissions - are looking at UNDP as a one-stop shop forsupport SIDP has been able to introduce a system of working throughLetters of Agreement (LoA)6 with the governments The LoA allow the twoparties to document their agreements in detail, provide for division ofresponsibilities and commitments, and give ownership – through managerialresponsibility – to the government partners in the management of theproject The LoA also allow re-planning of the activities quite flexibly inresponse to changing situations (more so than the larger project documentsdo)

The earlier evaluation documents and the (draft) final report submitted to EC

on 10th March 2014 give elaborate lists of achievements that were found to

be correct when visited the government offices and discussed withgovernment officers in Mogadishu, Hargeisa, and Garowe To cite a fewexamples:

a. The Somaliland Civil Service Commission and the SomalilandCivil Service Institute are at present functional governmentbodies The SIDP support contributed significantly to theseachievements,

b. In Puntland, SIDP has supported to reform Ministry of Labour fullyand they are now in a position to show a direction to other lineMinistries However, there is a need to reconcile issues with thecivil service commission in Puntland

c. SIDP could bring Puntland State University on the global map ofUniversities It is now a recognised centre for certification ofaccountants7

6 The LoA also have drawbacks as we will discuss further below

7 While it is recognized that ACCA courses may not be the most appropriate for public sector financial management, they do help to stimulate the emergence of a solid body of people conversant with world-class accounting standards, and the SIDP support to these courses has been instrumental in bringing the University to a higher level

21

d. Auditor General’s Office procedures development in Puntland isanother success story of SIDP

e. Despite frequent mentioning that SIDP is not sufficientlydelivering, the production of more than 50 different outputs(documents covering policy, legal framework, advisory notesguidelines on procedures, manuals, and other knowledgeproducts) during the currency of the project is quite anachievement8

f. Establishment of working relations with government offices inMogadishu, Garowe, and Hargeisa is another example of success.Government officers in these areas welcome UNDP support forcapacity development and institutional strengthening They haveexpressed hopes on future continued support from UNDP

g. Customs and Revenue collection systems in Bosaso portestablished and they are now working;

h. Provided infrastructure support to many agencies, e.g CSC, CSI

i. Collaboration with IOM QUEST-MIDA, established linkages withDiaspora, the bulk of participants in this program transitionedinto their regional governments

j. Internship programme is a success, and interns recruited throughthis programme are working at different levels in thegovernment The recruitment of these interns in such atransparent manner is unheard of Candidates, who neverexpected their entry into civil service without any clan or politicallink were surprised when they received offer letters based ontheir merit This has opened the doors to merit basedrecruitment system in the Government

k. International study visits organised, introduced internationalperspectives, and best practices Regional level liaison forexample with Ethiopia has set an example of south-southcooperation

l. Women employment in civil service and training increased

Capacity development includes development of an enabling environment forthe organisations and individuals function Recognising this aspect, SIDP hasbeen able to develop enabling environment by training parliamentarians in a

8 We recognize, as the previous evaluative reports have done, that the production of reports per se does not bring about change As we will discuss further below, the project has been insufficiently strong in following-up on these outputs However, without these outputs there would have been little to follow-up upon.

policy formulation, ICT, and legislation making Further, SIDP has placed twolegal drafters within Parliament, who reviewed and contributed to differentpieces of legislation

The table below shows the relations established by SIDP

South Central

 The Office of the

Prime Minister

(OPM);

 Ministry of Finance

and Planning; (now

split into the

Finance (MoF);

 Labour, Youth and Sports (MoLYS);

 Office of the Presidency;

 Puntland Civil Service Commission (PL CSC)

 Accountant General’s Office (AGO)

 Auditor General’s Offices (OAG),

 Ministry of Women Development and Family Affairs,

 Parliament.

 The Ministry of Planning and [National]

Development (MoPND);

 Civil Service Commission (SL CSC),

 Somaliland Civil Service Institute (CSI);

 Accountant General’s Office (AGO)

 Auditor General’s Offices (OAG);

 Somaliland Good Governance Commission

In South-Central Somalia, SIDP worked with the Office of the Prime Minister(OPM); the Ministry of Finance and Planning; the National Civil ServiceCommission (NCSC) and the Central Bank of Somalia In Somaliland, SIDPwas comparatively more active and worked with the Ministry of Planning andNational Development (MoPND), Accountant General’s Office (AGO) andAuditor General’s Offices (OAG); the Somaliland Good GovernanceCommission (GGC), the Civil Service Commission (CSC), and the SomalilandCivil Service Institute (CSI) Today the civil service institute Somaliland is amodel for Somalia; In Puntland, despite its minimal presence, the project hasbeen able to work with different government agencies Theinstitutionalisation of recruitment systems, civil servant registration systems

in the Ministry of Labour, Youth (MoLYS), and Sport are worth noting Today,MoLYS Puntland has a mandate, vision, mission, and goals It has anorganisational structure, and defined job descriptions The work in MoLYScan be an example of public administration reforms to begin with

23

Identifying National Youth and inducting them into public service throughinternship is another feather in the cap of SIDP However, the sustainability

of this approach is again debatable for the salaries of interns are reducedwhen they are absorbed in to civil service

2.2.2 Project Shortfalls

As discussed in the paragraph on project successes, the SIDP supported theinstitutional development South-Central Somalia, Puntland, and Somaliland.However, SIDP has taken the blame of being a dysfunctional project For anyproject of this magnitude functioning in a volatile environment, blame ofbeing dysfunctional is understandable Complaints on reporting, relationsbetween the project staff and project leadership are obvious in any projectunless they result in total crash of the project management and delivery ForSIDP it is not the case The entire operational environment includingrecruitment of internationals for UNDP Somalia, a conflict country was achallenging situation for this project Staff shortage, weak reportingrelations, non communication with the field staff, non release of promisedfunds on time, delay in recruitment, non provision of required technicalexpertise, remote management and non record keeping are some of theoperational shortfalls that have been brought to the notice of the evaluatorsduring the evaluation

For example:

a Relations with other projects, Govt, and Donors: There is no

strong evidence that the project has established linkages with otherprojects, with the exception of some of the activities under Governanceand Rule of law programme under which the project was operating

b Ownership issues, supply orientation vs demand driven: These

are the sentiments of the stakeholders Who has to own the reformsprocess is still a challenge Additionally, the major issue came beforethe evaluators is the issue of the project being a supply driven projectnot a demand driven project

c LoA system was not clear The covenants of contract were very

lengthy and not understood by many Secondly, there went a messagethat LoAs would be stopped;

d Follow up on issues missing: There could have been follow up on

the actions initiated particularly on the project documents developed,systems established, and support provided

e Un-sustainable QUEST-MIDA salaries: Government feels that the

salaries paid to Diaspora staff are very high and not sustainable.Secondly, in case of Diaspora, there is no link between the work theywere doing in a foreign country and the duties assigned here in thepartner organisations of the project

f Support to institutions - not institutional development has

created a capacity substitution environment not capacity development.All these shortfalls are predominantly environmental not specific to SIDP Theprojects working under the circumstances obtaining in the country likeSomalia, and a project with limited resources with instability in thegovernment will have such generic shortfalls

25

3.0 Evaluation of three themes

We now discuss the findings on evaluation of three thematic areas taken up

by the project for implementation;(1) public administration and civil servicereform; (2) public financial Management; and (3)development planning andaid coordination SIDP supported the different governments for all the threeactivities The ToR for evaluation seeks to evaluate the actions initiated bySIDP and activities taken up by the governments The finding of evaluation ofthese three thematic areas would feed into future support to institutionaldevelopment programming

3.1 Public Administration and Civil Service Reform- Overall findings

The terms Public Sector and Public Administration are used inter changeably

in the project document Without going in to their etymology, we go by thespirit of the term for our evaluation

At the cost of repetition, we should not forget the circumstances under whichthe project operated Change of seven prime ministers, seven cabinets, fourproject managers, understaffing of the project, not recruiting the people ontime, remote management- all these have been major challenges for projectdelivery as planned

Neither the public administration experts nor the public financialmanagement experts fully agree that the public administration reforms arefully possible through support to public financial management PFM can bepart of PAR but not PAR in itself Public Administration Reform (PAR) is an

UNDP and European Union Review Reports

The report produced by UNDP on SIDP and submitted to EU on 10 th March 2014 is very comprehensive and candid Evaluators do not like to dispute with the progress, successes, and shortcomings, enumerated in the document, that were with special reference to public administration and civil service reforms in Somaliland, Puntland and South-Central Somalia Any further comments or reporting would be duplication of the report It will be writing again what has already been written, albeit, in the language and style of evaluators Therefore, it is decided not to duplicate the report.

The evaluation report of EC of 2012 is not disputed by the evaluators The EC report has considered the situation prevailing during the period under their evaluation This evaluation has an extensive coverage of activities taken up at all the tree places under civil service reforms The evaluation has very minutely reviewed the documents produced under SIDP support, and made extensive and elaborate recommendations UNDP has taken cognisance

of observations made in the EC report with specific reference to public administration and civil service reforms Nevertheless, institutionalisation of public administration reforms still needs

ongoing process in any government It emanates from political commitmentsfor public service delivery It breaks inertia in the service delivery systems,introduces new business processes, sharpens delivery tools, establishespublic accountability and transparency systems, improves financialmanagement, requires human resources undergo changes in skills, attitudes,and knowledge level in accordance with changing public demands for troublefree services Therefore, it is very challenging in any state, more so in fragile,conflict, and post conflict countries In countries where systems weredestroyed, the first step is institutionalisation of public administration orsetting up public service delivery systems rather than public administrationreform Reform is introduced in the existing systems Generically, the firststep in establishment of public administration systems and make themfunctional Reform comes at a later stage when they became redundantdue to changes in services requirements and behaviour of citizens It is aprocess and a continuous process.9 Focus could have been more onestablishment of public administration in the regions where it was not fullyestablished, and reform in the regions where it is comparatively developed10.Generically, Public Administration begins with defining a mandate for thegovernment Identification of functions of the government, enlisting publicservices, designing organisational structures linking up functions,classification of authorities, and hierarchies, defining reporting lines,describing the jobs, roles, responsibilities, and recruiting people, trainingthem and placing them in correct place at correct time are some of thegeneric requirements of PAR In the present context, mustering politicalsupport, putting together the administrative structure, building on theavailable bases, institutionalising the processes, and developing civilservants and sustainable benefit and entitlement packages is a tough taskconsidering the political and economic environment, and takes time Not all

of PAR activities may be taken up at the same time but focus on systemicdefects with a systematic approach is definitely required

The Somali Institutional Development Project document has identified theweaknesses of PAR issues that need to be addressed through the project Itenlists policy and legislative systems and processes related to developmentmanagement and democratic good governance, poor macro-economic andsocial statistics, nascent budgetary and procurement processes, incapacity

9 For example in India, the public administration reform commission was established in January 1966, and the reform is continuing even today

10 For example the systems in Somaliland are better than they are in Puntland, and South Central Somalia

27

to produce and audit annual financial statements, and the absence ofParliamentary and Civil Society Organisation (CSO) oversight of budgetaryprocesses as weak areas.

All the four pillars of support envisaged in the project document are part ofoverall public administration reform areas This has been rightlyacknowledged by the project document in several places However, a PublicAdministration Specialist, to lead the process of public administrationreforms, institutional development, and civil service capacity developmenthas been appointed only during late 2012 during the extended period of theproject Although, the project document was signed in October 2008, and theproject was launched in January 2009, no serious attention was given topublic administration until the new expert joined, based in Hargeisa,Somaliland Even though it was repeatedly said in the project report thatdifferent regions require different approaches for reform, ultimately theproject has taken up a single approach for all the regions The proposedactivities for all the three regions are similar and generic It can be argued byproject formulators that at a state building stage, only generic approachesare adopted

The interventions proposed in the results framework for all the regions havebeen:

Drafting priority laws on government operations, support the governmentuse ICT, inter-ministerial coordination, communication& public consultationmechanisms, civil service reform, civil service training, macroeconomicpolicy planning, budget formulation and preparation, budget execution,internal audit, parliamentary oversight, procurement, accounting andreporting, infrastructure and public support These areas were taken up bothfor institutional development and for individual capacity development

The PAR initiatives and institutional development activities taken up by SIDPhave shown both positive results and delivery challenges Many of theactivities of SIDP to support public sector and civil service reforms arecontinuation of past projects like SICAD

To sum up public administration and civil service reform efforts:

In South Central Somalia, UNDP has worked with federal government allthrough its transitions and changes SIDP has been able to establish itsidentity in South-Central Somalia through working in Mogadishu despite

security concerns more volatile than in Somaliland and Puntland At thebeginning of project, and before the merger of other projects into one, UNDPhad provided some equipment, a vehicle to the civil service commission inMogadishu However, they were destroyed during the clashes; thereafter thegovernment moved to Nairobi for few months SIDP has again providedchairs, tables, desks, computers, and some technical advice to the civilservice commission It appears that SIDP tried to push more activities during

2013, but out of seven activities promised to the national civil servicecommission at Mogadishu during 2013, only one activity-providing technicaladvice on civil servant information management was partially completed

In Somaliland, SIDP worked very closely with civil service commission, civilservice institute, Ministry of planning, and Auditor General’s office Although,there is more to be done, much has been done in the Civil ServiceCommission and Civil Service Institute The public administration reformexpert positioned in Somaliland has established himself well as a mostsought after person by the government for support and advice in the area ofpublic administration reform

In Puntland, SIDP worked with Ministry of Labour, and changed its style offunctioning altogether It is an example of public administration reform start

up Office of the Auditor general is another example where audit systems arebeing formalised Supports to Ministry of Finance to enhance revenuegeneration at Bossaso port etc are some of the examples of SIDPS exemplaryperformance

Support through providing members of the Diaspora to the governmentthrough QUEST-MIDA has not given the desired results in all the three places.Sometimes, it has become a burden for the government as the personsrecruited through QUEST-MIDA were very expensive and did not matchtechnical requirement of government Neither, many of them did they havework experience relevant to the government systems and procedures In the

words of Deputy Minister MoPIC Somalia at Mogadishu “QUEST-MIDA recruits

have come to Somalia as summer interns to earn some quick money and go back There was no sustainable contribution from them”

3.2 Public Financial Management

The activities undertaken by SIDP in the area of Public Financial Management(PFM) were broad in scope, and were targeted at addressing the PFM needs

29

in each of the 3 regions With few players in the field of PFM at the time ofthe initiation of SIDP, there was an overwhelming need for assistance in theestablishment, and reestablishment of PFM systems through activities inboth institutional development and capacity development.

The response from SIDP included: the sponsorship of training programswhich lead to an internationally recognize ACCA certificate in accounting, theinitiation of technical assistance in accounting, auditing, procurement,budgeting, and the provision of advisors and staff in Ministries of Finance (aswell as associated PFM institutions)

Taken alone, PFM would have been a technical field able to consume theentire resources of SIDP However, PFM is not simply a set of technical rules,regulations, and forms to complete; PFM is as much a mind-set as is theconcept of democratic governance itself This means that PFM requires long-term, sustained, and incremental involvement For example, while anaccounting item code (chart-of-accounts) can be drawn up in a relativelyshort time, the task of linking these new codes to past accounting is a majorundertaking, as well, many thousands of government employees play somerole in the handling of financial transactions This requires substantialtraining efforts, efforts which may require dozens/hundreds of trainingsessions and years to accomplish

SIDP attempted, certainly through good motives, to fill many of the PFMvacuums where they appeared And the requests were many andsubstantial If there can be any overall critique it can be that:

3.2.1 SIDP attempted to do too many things in PFM

As a result of taking on too many tasks, the inputs devoted, the outputsdelivered, and the outcomes desired could not be fully delivered In sum:

3.2.2 SIDP results in PFM were only partially achieved

If anything, the appetite for PFM has been raised in all three regions This initself is a positive achievement Expectations have been raised that UNDPwill play a role in the future in PFM, at least the desire for further resourcesand technical assistance is there So not only do the governments have anappetite for PFM reform, they are hungry for it However, as discussed below

it is important to not underestimate the resources required to enter intomajor institutional development reforms in PFM

The attempt to do too many things had a negative impact on what was done.For example, extensive manuals were produced and the implementation ofthese manuals was only partially completed Many “good starts” weremade, but the follow-through was often not able to be completed within thetimeframe of the project.

This leads to several points which impacted SIDP, and as will impact anyfuture PFM programming:

 PFM requires specialists in fields which are very diverse, i.e an expert

in accounting may not be an expert in budgeting, as well specialtiessuch as auditing, procurement, internal audit, control, taxation andtreasury are each distinct functions,

 PFM reform takes sustained efforts (several training programs is notsufficient to achieve behavioural changes in PFM practices),

 PFM reforms require very detailed follow-up, advising, and mentoring

on the ground, i.e the work needs to be undertaken in the offices ofthe practitioners themselves,

 PFM touches onto many additional spheres of governance, i.e.relationships between ministries, relationships between levels ofgovernment, relationships between the parliament and the ministries,

 PFM requires incremental introduction, i.e to introduce new budgetformats requires one step at a time, since many of the requirementsare sequential and require the buy-in and political will of politicalleaders to fully implement,

 PFM is in itself an intensely political field, the control and allocation offinancial resources adds many political aspects to the work Thisinfluences peoples behaviours, including the willingness to share data,

to share responsibility, and to report in transparent manner

The efforts in PFM in the end are not without merit The involvement ofUNDP in PFM raised the awareness of the current status of PFM in each of thethree regions SIDP involvement pointed to the outstanding needs, andprovided some positive first steps in addressing PFM issues If anything, theability for SIDP to openly engage in PFM is a positive step, and shows anopenness, willingness, and commitment to transparency by thegovernments Even in Somaliland where the PFM engagement was

31

contentious (see below), the government still looks to UNDP for possiblefuture engagement in PFM related activities.

3.2.3 Public Financial Management - Findings specific to Somaliland,

Puntland, and South-Central Somalia

In Somaliland, the situation came about whereby UNDP has been “invitedout” of participating in PFM activities by the Ministry of Finance (MOF) TheMOF, in its wisdom, invited all donors and TA programs “out” This was due

to the contradictory advice being given and the quality of TA provided TheUNDP was not invited back “in” to reengage and provide support, and wasquite directly requested to halt providing assistance to PFM, even to halt itsefforts to ministries other than MOF (this was MOFs point of few, notnecessarily the viewpoint of the other agencies or ministries)

SIDP over-reached its capacity to provide the assistance agreed in the LOAswith partner ministries Particularly the LOA with the MOF over-promised Tofulfil the commitments (made by both sides in the LOA) would have requiredthe efforts of 3-5 years of separate project teams However in theenthusiasm of both the MOF and SIDP, unrealistic timeframes of 3-6 monthswere established As can be seen by the examples below (from the MOFLOA) extremely ambitious performance objectives were established (thesewould have been ambitious in any setting, let alone in Somalia):

 Macro-economic unit officially launched and operational and awarenesscreated about its mandate

 A new Public Finance and Accountability Bill drafted and made readyfor presentation to parliament

 Compile a list of tax payers and introduce tax segments- large taxpayers and small tax payers

 Propose to the government a customs tax and administrationprocedures that harmonizes the customs valuation and administrationprocedures with regional practices

 GFS 2001 (economic classification, first, functional classification later)adopted for presenting the budget Note: Government FinancialStatistics (GFS 2001) are not budget codes but accounting codes

 Make operational the new fiscal and tax policies through the adoption

of new tax laws and the introduction of fiscal code or fiscalaccountability measures

Of a particularly positive note for PFM development is the National

Development Plan developed by the Ministry of National Planning and

Development (MoNP&D) It is a document that is comprehensive in nature,includes the plans for all ministries and government units and includes theplans for investment by donors Some sectorial allocations (or suggestions)

of financial resources are made within the Plan In discussion with the staff

of the MoNP&D a copy of the annual budget was requested Whenquestioned “how is the Plan and the Budget joined?”, the answer was: “theyare not” The budget format in Somaliland is the same as in Puntland andSouth-Central Somalia; it is a line item budget based on item codes (i.e howmuch stationary will be procured) However, the Plan is designed along verywell defined programs In essence the Plan is more of a budget, than theBudget itself

Issues arose as well in Somaliland with the administration of LOAs Changes

of staff in both the government and UNDP meant that LOAs fell out of theinstitutional memory The formal follow-up, and reporting against the goalsset-out in LOAs was often lacking As well, while the LOAs divided sums offunds in blocks between various activities, the budgets for the use of thesefunds were not adequately made (i.e the budgets were divided into grossamounts per activity, but the budgets were not build from the ground-up toarrive at the true needs of the activities This lead to the poor practice ofspending for one activity, but using the budget designated for anotheractivity Thus the accounting for both activities did not reflect the actualcosts incurred At the best, this is a poor accounting procedure)

Some of the issues to be aware off and to avoid/correct in future activitieswere highlighted by the government officials interviewed While some of therecommendations are of a technical nature, the majority are of amanagement nature It is clear, because of the close relationship of anumber of the ministries, that the advice and recommendations providedstems from their experience in working with SIDP over time Among theuseful comments and suggestions gained in Somaliland are:

 The “vision” of LOAs were lost when management staff or governmentstaff changed, there should be a formal process of “brining LOAsforward” whenever a major staffing (project or government) occurs,

33

 There are limitations to what the government can do, and what it canabsorb Be cautious to not overly tax the ability of the governmentagencies engaged,

 Both the ACCA and CAT accounting trainings were appreciated (CAT isthe basic level, leading to the advanced ACCA modules), however, forpractical purposes consider adding the CIPFA program which is moregeared towards public administration needs,

 Consultants left behind reports, not capacity The attitude andapproach of “we will do it for you” is incorrect from the perspective ofinstitution building,

 The support was incomplete, we train someone, but then they need acomputer in their office to do what they were trained in, and thisresource was missing,

 When communications are poor, adversarial relations will occur, bettercommunications are required on the implementation stages andprogress of LOAs,

 People with experience and expertise are needed, not just withacademic degrees,

 Embed advisors in the offices in the ministries, do not house themoutside,

 Quality control is required, the consultant may produce a document,but it may be of poor quality and not useful,

 The goals must be clear and not vague,

 “Capacity development does not mean just a consultant working intheir hotel on a laptop, a more complete package is required” Was astatement heard repeatedly throughout the process of meeting withPFM counterparts This reinforces that the government officials wereeager for more, and more interactive and intense relations with thetechnical assistance advisors The counterparts desired more time andefforts of the technical advisors to sit down and work side by sidewith local counterparts

 In Puntland, the SIDP interventions in PFM related activities weregenerally acknowledged to be successful/partially successful (i.e more

activities could be used) As included in the SIDP Prodoc, a very widerange of PFM support activities were entered into Among the PFMactivities conducted in Puntland are:

 ACCA program (107 government officials enrolled),

 CAT program (200+ government officials enrolled),

 Audit training

 TOT training

 Strategic planning training

 Draft audit bill

 Draft audit regulations

 Tender Board, Standard Bidding training

 Standard auditing procedures, audit planning, and audit checklist

 Customs assessment and regulations

UNDP provided much of the capacity development assistance throughsecondary contractors; KPMG and EY While acknowledged by thegovernment as having provided generally high quality results, there appear

to have been several major deficits in the PFM TA provided:

 The deliverable products tended to be manuals which were heavilytheoretical in nature, of a volume which was overwhelming, and thedocuments were provided in English copy only and without translation,

 TA was of uneven quality, the duration was often short, and TA wouldarrive to only provide an output, i.e.; a training program of severaldays No residual follow-up or coaching was provided,

The Auditor General related that the SIDP provided the first support of anykind for training programs

The Ministry of Planning characterised the SIDP assistance as: “Without SIDP,

we couldn’t do anything” But of course, a good start requires a good

follow-up, by both the government and UNDP A follow on quote illustrates this:

“We made the Plan, now we have to implement it and we don’t have theteam to do it”

35

The Auditor General noted that communications with UNDP were veryeffective As well a statement was made that had an interesting caveat:

“What the consultants delivered did not match with the TOR, they did more!

We got more value than what was spent” But continuing on, it was addedthat; “through SIDP we got only 5% of what was needed” So SIDP wasrecognized as a good start, but more activities are required in the future

In South-Central, as has been noted the frequent change in governments hasmeant that there is a knowledge gap within the government itself regardingthe various LOAs in effect Even the Ministry of Finance and Planning wassplit into two separate Ministries, and there was no correspondingrealignment of the LOAs So neither the government, nor SIDP, wassuccessful in updating and maintaining the validity and application of theLOAs

Because of the special difficulties in working with Government officials inMogadishu, few data points related to the performance of PFM assistanceprovided by SIDP were able to be gathered The changes in government andlack of continuity planning on the part of both SIDP and the Government areevident There did not appear to be clear understanding of the status ofLOAs, what was agreed, what was “promised”, and what is the current state

of the agreed deliverable services and items, (note: some governmentofficials have been in their post for 4 months) In particular, the AccountGeneral (new to the post with the last change of government) expressed hisdissatisfaction with the performance of KPMG Quoting: “KPMG delivered anunwanted manual, it is 804 pages in length, and nowhere in any singleparagraph is the word “Somalia” This is not what we need, we need toprogress from the starting point from where we are now, to the next step.”

As a side note, it can be observed that World Bank has provided an excellentSenior Policy Advisor & PFM Reform Coordinator to the Ministry of Finance,and that the World Bank has taken solid steps in establishing and is in theinitial steps of operating an electronic financial management system

3.3 Development Management and Aid coordination

3.3.1 Project Design

Evaluations ask pertinent questions to generate analysis and debate forimproved programming Foremost of those are the assumptions, objectivesand analytical approach underlying the programme being evaluated, all ofwhich define the theory as to how the programme design will lead to thedesired results.11

Assumptions

The theory of change behind capacity development of governmentinstitutions is dominated by certain assumptions that often relate to will andincentives Who is interested in the change? What incentives do donorinterventions create when supporting capacity development? Whodetermines the ultimate agenda? How does a development agency addressgovernment ownership when its priorities fail to meet the donor requirementfor effective methods?

The prevailing view seems to be that donors can achieve objectives that help

a country obtain a certain development profile with enough interventions,time and money However, the evaluation of development results suggeststhat this may be a false assumption If the donor concepts about how thesociety functions are not recognizant of the existing system and interests,there is no reason to think that the same methods, repeated over time, willbecome more effective Some innovative thinking is required, reflecting thecomplexity of the setting, instead of trying to apply methods that might besuitable in a stable middle-income country

Compatibility of Objectives

If SIDP’s aim is to build the capacity of the government to play the primaryrole in essential service delivery, then the concern is to assure thatcapacitated staffs remain in the Ministry However, once staffs receivetraining and experience, many of them leave to take a position with aninternational agency.12 Therefore, the capacity development measures thatfocus on training of staff actually tend to leave the Ministry only marginally

11 For more effective evaluation utilization, those reading the evaluation should make note of issues raised, suggestions, points of debate and discuss them as a group exercise in

capacity building at the systems level Knowing the limitations of their own system, they can recommend a way forward This evaluator does not offer a list of recommendations, but expects the reader to derive them from the report

37

more functional In other words, the most prevalent capacity developmentstrategies can leave a Ministry less functional Clearly, training of individualsdoes not mean they will be able to incorporate that into their work, nor have

an impact on the system operative within their ministry That requires asystemic strategy with longer-term assistance tailored to the specificity ofthe setting

Some agencies have taken to offering a salary stipend to Ministry workers

13in order to reduce the gap in pay between government and developmentagencies This can help to retain capacity in the short term until either theGovernment collects enough revenue to raise salaries to an attractive livingwage or until international agencies reduce salaries to approach governmentlevels The ideal would be to agree a sum that provides a pay scale thatcould in a matter of years be sustained by the government Providing astipend only assures results as long as the payment is made leasedcapacity

SIDP used highly paid short-term international or embedded diasporaconsultants The short-term consultants provided some valuable outputs,guidelines and documents, but were not present long enough to incorporatetheir contributions into the system International consultants with technicalexpertise indicate they need to provide long-term input, of six months to ayear, even for some basic procedures.14

Diaspora consultants create some tensions with other governmentemployees, who think they know the system better and could often do thesame work—but do so with a fraction of the salary It is difficult to see whatincentive a diaspora consultant would have to transfer capacity, as he would

be eliminating his own job Incentives have to be examined in order toassure the capacity development strategy yields the desired results Indeed,almost all of the diaspora consultants highly paid though the QUESTS-MIDA

12 While there was no way to ascertain how often this happens, staff routinely mentioned it

as the path to earning more money outside of government A short course, however, does not add much value, unless the person is already quite capable and well placed in the system.

13 Salary scales between ministries For instance, Ministry of Finance staff in Somaliland earn almost twice as much as Ministry of Planning staff Stipends may double the salary, such as the stipends paid by UNICEF in the health and education sectors in Puntland While a living wage in Puntland was estimated at $170/month, government staff may earn as little as $60/ month.

14 Ministry of National Planning in Somaliland confirmed this, as did an M&E consultant whose contract were short term.

programme of the UNDP and IOM have remained on although theGovernment cannot afford to pay them—in the hope of another contract.

3.3.2 Project Analytical Framework

Developing the Capacity of Institutions

Capacity development, of necessity, begins with the provision of materialcapacity through office developments, furniture and equipment.15 Then itmoves to training staff-the human resource capacity that gives an institutionits functionality However, individuals with training may then be a morevaluable asset and may leave the government sector in order to pursue ahigher paying job with a foreign agency As long as development efforts aredominated by foreign agencies, there will be higher paying jobs outside ofthe government, and it will be difficult to maintain human resource capacitywithin a ministry16

While the loss of trained staff may limit the capacity development of theintended institution, it does not mean the capacity development effort failed.The trained personnel are probably adding value to the nationaldevelopment effort in their new employment, in a broader distribution, eventhough they are not concentrated in the target ministry Rudimentarytraining would not be that valuable, but gives them enough visibility to behired by a development agency with a functional system that can developtheir skills and give them experience This lays the foundation for expandingthe development capacity of the country, rather than the human resourcebase in a single ministry

Tracking their contribution in their new place of work would provide a morevaluable measure of overall capacity development in a country with suchvast needs It should ultimately be quite effective over time to train dozens

of staff interns in the Ministry of Planning, for instance, in skills such as

15 There are four main types of capacity to be developed: material, human resource,

financial and systems Too often the focus is on the individual, who can rarely change the system in which he operates.

16 While this evaluation relied on self-reporting and not independent verification, many people said that a government worker who is trained would use that skill to seek a higher- paying job elsewhere, and who would blame him?

39

planning, financial management, M&E and then track where they took thoseskills and to what effect.17

The Ministry of Planning can be the access point for providing basic capacityfor the core development sectors, and develop monitoring skill as some ofthose staff move on to work in a range of agencies where they can gainpractical experience As development funding shifts from internationalagencies to direct implementation by Ministries, many of those staff personswill migrate back to government employment, bringing with them the richand diverse experience of working with international organizations and theirvaried outlooks This approach would constitute a five to ten-year capacitydevelopment effort What is lacking is a mechanism for these trainees toprovide ongoing input to the Ministry about what they have learned and how

to integrate the lessons of their new job with the national development plan Such a tracking and feedback mechanism would not be difficult to establish.Perhaps twice a year they could meet up as part of the sector workinggroups to share what they have learned and help to inform the government’spartnership and cooperation with aid agencies

Development Planning

The Paris Declaration and Accra Agenda affirm that self-determination shouldguide the development process The Busan agreement behind the New Dealfor peace-development in fragile states outlines pillars of priorityinterventions—a donor-determined agenda The New Deal offers the basis forthe Multi-Partner Development Funds that will fund various developmentplans

There are multiple Somali development plans that could serve as the basisfor action: the Somali Compact with the Special Arrangement for Somaliland,The National Development Plan of the Ministry of National Planning andDevelopment (MNPD) with the Somaliland Vision 2030, the PuntlandDevelopment Plan, the new Development Plan now being sponsored by theMinistry of Planning and International Cooperation of the FederalGovernment In meetings with the government ministries dealing withdevelopment planning and aid coordination, all referred to their own nationalplan rather than the internationally supported documents, as they spokeabout ownership.18

17 This could be the role of a long-term consultant placed in the Ministry to train waves of graduates.