An-Instrument-for-Measuring-the-Quality-of-Enterprise-Architecture-Products

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/242752354An Instrument for Measuring the Quality of Enterprise Architecture

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/242752354

An Instrument for Measuring the Quality of Enterprise Architecture Products

Article

CITATIONS

2

READS

469

4 authors, including:

Some of the authors of this publication are also working on these related projects:

Relationship between EA maturity and the quality of IT investment decisions View project

XAI in the financial sector View project

Martin van den Berg

Hogeschool Utrecht

28 PUBLICATIONS   313 CITATIONS   

SEE PROFILE

Rik Bos

Open Universiteit Nederland

38 PUBLICATIONS   395 CITATIONS    SEE PROFILE

Sjaak Brinkkemper

Utrecht University

443 PUBLICATIONS   9,663 CITATIONS   

SEE PROFILE

An Instrument for Measuring the Quality of

Enterprise Architecture Products

Elise Veltman-Van Reekum (kluwer B.V.)

Marlies van Steenbergen (Sogeti Nederland B.V.)

Martin van den Berg (Sogeti Nederland B.V.)

Rik Bos (Universiteit Utrecht)

Sjaak Brinkkemper (Universiteit Utrecht)

On average Dutch Enterprise Architecture products are attractive and provide guidance to the enterprise in times of change However, they rarely address all issues completely Only 50% of the Enterprise Architecture products are judged readable The Enterprise Architecture products are very explicit about their status but do not mention a clear time span for the architecture to be valid or how the architecture should link to the existing enterprise These are some of the results of a study to assess the quality of Enterprise Architecture products, we conducted in

2005

For many, the term Enterprise Architecture conjures up visions of colorful models and fat documents with guidelines These products of the Enterprise Architecture process are indeed its most tangible result Most architects can intuitively tell which models are the most readable and which guidelines are the strongest but that intuition is not objective enough to produce an independent measure of the quality of

an architecture product Until recently, no such measurement existed This paper explores the concept of Enterprise Architecture product quality and how to measure

it in a scientific manner

Enterprise Architecture products

In practice, quality measurements are meant to assure “with a degree of confidence that the products are depicting the necessary information in the right way” (Galin 2004) So products can be adjusted in time to negate the risks involved in not meeting the goals of the product These risks are usually financial in nature

Richardson et al identified in 1990 the lack of consensus concerning what a good architecture entails (Richardson 1990) Within enterprises, this lack of consensus can impair the effectiveness of the Enterprise Architecture (EA) By building the architecture with quality assured products, the discussion concerning the EA can be focused on its content instead of its quality In other words, the quality measurements are meant to create consensus about the concept of ‘a good architecture product’

In this paper an instrument for measuring the quality of EA products is presented For understanding all details of this instrument we kindly refer you to the technical report “Determining the Quality of Enterprise Architecture Products” (Veltman - van Reekum 2006)

When discussing products of any kind inevitably their relationship with processes comes up, since processes create products However, each concept is very different

in nature and therefore requires a different perspective and approach to judge its quality This research was conducted in close relation with a similar research into a quality instrument for Enterprise Architecture processes (Bent 2006) In this way both instruments are based upon a similar thought pattern on EA quality, enabling architects and consultants to use both instruments simultaneously in the knowledge that there is no overlap between the two instruments

Definition of good EA product quality

“Enterprises are constantly subjected to internal and external changes (Mintzberg 1992; Bernus 1997; Jonkers 2004) One way to deal with these changes is to be integrated (Mintzberg 1992; Wagter 2001; Charpulat 2003; Giachetti 2003; Hoogervorst 2003) and agile (Wagter 2001; Bernus 2003; Hoogervorst 2003) Enterprises can use Enterprise Architecture as a way to achieve integration and agility in a coherent and consistent way (Rijsenbrij 2002), through EA products created by the EA development process Therefore, the quality of an EA product is its ability to aid integration and agility in a coherent and consistent way.” Figure 1 graphically depicts the above described principle on which the instrument is based

Figure 1: providing integration and agility in a coherent and consistent way Agility is defined by Dove et al as “the ability of an organization to adapt proficiently (thrive) in a continuously unpredictable business environment” (Dove 1996) The link between architecture and agility is addressed by The Open group in their answer to

‘Why do I need enterprise architecture?’ They describe how a good EA empowers individual enterprise parts, such as a business unit, to innovate safely in their pursuit

of competitive advantage (The Open Group 2003) (FAQ) This is clearly a form of agility The most compelling argument for the use of architecture in creating agility is made by Hoogervorst He states that architecture can take away inert characteristics that limit the enterprise in taking prompt action (Hoogervorst 2003)

Although agility is a necessity in effectively dealing with change, the enterprise does run the risk of loosing internal cohesion Chances are that the empowered enterprise parts engaging in individual change processes will indiscriminately move in different directions This can be avoided by relating all the different changes going on in the enterprise to each other and to those parts of the enterprise that are unchanged We call this integration

The level of integration of the enterprise is influenced by the relationships between the domains within the enterprise, as demonstrated by the domains depicted as gray blocks in Figure 2 Integration is also influenced by the relationships with the environment of the enterprise If the relationships are rigid in nature, change can be very difficult to achieve It is therefore not desirable to blend domains together unrecognizably This tends to create rigid and slow enterprises, which have trouble incorporating new developments For example, if a product domain is tied in closely with a technical domain then change in one highly affects the other If the effects are

unclear then this impairs agility greatly but having no relationship makes each domain ineffectual in achieving the enterprise mission Integration of the enterprise should therefore be aimed at creating

recognizable domains that related together are more responsive and effective than each element in isolated operation As inspired by (Kosanke 1999)

Figure 2: integration

EA attempts to achieve the goals of agility and integration in a consistent and coherent way (Richardson 1990; FAWG 2001; Wagter 2001; Rijsenbrij 2002; Bernus 2003; Hoogervorst 2003) Consistence and coherence are concepts that are closely related and are usually understood to be the same thing In this paper, the meanings found in Merriam-Webster’s dictionary are used rigidly to define the concepts individually The dictionary defines coherent as ‘logically or aesthetically ordered’, in other words the EA product is logical Although aesthetics are a part of EA (Rijsenbrij 2004), in the case of change the logical aspect of being coherent is the most alluring Logic has a deliberate, deductive and formal quality Consistency is stated by Merriam-Webster to be “free from variation or contradiction” In periods of change, it

is important that people can rely without question on the guidance that is offered to them

Definition of EA product

To understand the quality of EA products it is important to have an understanding of what such a product entails Many literature sources state the different formats that these products can have Such as “…a series of principles, guidelines, drawing, standards and rules…” (Boar 1999) or “ graphics, images and/or narratives” (FAWG 2001) or “…principles, separated into rules, guidelines and standards sometimes recorded into patterns…” (Rijsenbrij 2002) and many other sources (Cook 1996; Bernus 1997; IEEE 2000; Wagter 2001; Hoogervorst 2003; The Open Group 2003; Arsanjani 2004; Schekkerman 2004) When closely analyzing the nature of all these different products two properties were found One, all products have content that is related to different enterprise domains Two, all products have some sort of format

A decomposition of the nature of EA products can be created by combining these two points with the already stated purpose of EA to create agility and integration

Figure 3: decomposition EA product From this decomposition, a new definition for products was formulated because none

of the existing definitions could break away from its own particular development method or terms associated with format

If the definition is biased towards a certain type of product then the quality measurements for that product will always be higher than for products not fitting the definition This is scientifically very unsound

For this research, therefore, an EA product is defined as a set of tangible enterprise domain reflections in either language or visualizations Complexity in scope often demands to capture the subject of the product in a set of reflections to see the complete picture Even though reflections sound elusive which is very much the opposite of tangible, a choice has been made for the term reflections because, as Merriam-Webster’s dictionary says, they are ‘considerations of some subject matter, idea, or purpose’ As stated, many definitions describe EA products in terms of format, such as guidelines or graphics A graphic is not written and a guideline is not

a picture EA products usually contain a mix of language and visualization formats

In this paper Enterprise Architecture is understood to be the products that guide in the design and evolution of the enterprise and its relationships, to coherently and consistently aid the enterprise in achieving integration and agility

Logically, this means that the products guiding the design and evolution of the particular enterprise must adhere to the definition to be of a good quality Therefore the quality of an EA product is the totality of attributes that bear on the EA product’s ability to aid integration and agility in a consistent and coherent way

Measuring quality

The definition of EA product quality suggests that there are four dimensions that make up that quality Namely that which the product is meant to aid: Integration and agility of the enterprise and the way the product must be: coherent and consistent These four dimensions are placed in a 2 by 2 model that defines their relationships

Table 1 Quality model of an EA product

Logical unity

Is there logic in the way the product unites the changing and unchanging parts of the enterprise?

Logical adaptation

Is there logic in the way the product guides the adaptation of changes within the enterprise?

Reliable unity Can the enterprise rely

on the proposed integration in the product?

Reliable adaptation Can the organization safely rely on the guidance in adapting to changes as offered by the product?

The use of such a model is inspired by the AIMQ method for information quality which uses the method of a 2 by 2 model (Lee 2002)

To answer the questions in each quadrant we developed an instrument Attributes are used to measure the level of quality in the product; together they form the answer to the questions A set of quality attributes was gathered from methods in the fields of software (Zeist 1996; Moody 1998), information (Kahn 2002) and data (Wang 1996) The comprehensive list of attributes in the developed products together with definitions can be found in Appendix A This set of attributes is by no means exhaustive It is a starting point for the instrument, but it is expected to be a promising set because of its basis in such different proven quality methods, all based

on the theoretical framework that is presented above and because they were subjected to a sound scientific test (Veltman - van Reekum 2006)

Plotting the attribute set on the quadrant model the instrument for measuring the

quality of an EA product looks as shown in table 2 Two attributes describe the

product as a whole namely the completeness and assess-ability of the product They

are therefore applicable to the overall quality model

Table 2 the quality model filled with attributes

Logical unity

Logical adaptation

Reliable unity

Reliable adaptation

How do you use the attributes explicitness and stability to measure if the

organization can safely rely on the guidance offered by the product? Well you need

an auditor who decides what the level of explicitness is But you don’t want him to

simply decide for himself You want to give him guidelines which describe

explicitness These guidelines are called items in our instrument For instance in the

case of explicitness an item is: “The product describes its own status” or when

judging stability an item is: “The consequences for other solutions of changing a

solution are described”

The totality of the items describe the ideal state for an EA product and make up the

scale of an attribute The term scale is used to refer to all items within an attribute

(Flynn 1994) It is up to the auditor to state whether or not they agree that the item

is applicable to the product under scrutiny The auditors can express the level of

quality by choosing between ‘totally disagree’, ‘disagree’, ‘agree’ or ‘completely

agree

By attaching weights to the four possibilities, the percentage at which an attribute is

present in the product can be calculated A ‘completely agree’ for all items of

‘explicitness’ means that ‘explicitness’ scores 100% present in the product When a

product scores a 0% on stability this means that the attribute is not present in the

product at all (I ‘totally disagree’)

Situations can arise that make items or even whole attributes not applicable to the

product in question This has been taken into account in the instrument By choosing

n/a, the instrument will not use the item in the calculation of the attribute % level

Appendix A also contains the items per attribute

- Completeness

- Assess-ability

Results and Conclusion

We tested the instrument thus developed, on 26 EA products from 23 different

organizations Though this test was primarily meant to check the validity and

reliability of the instrument, we were able to present the participants with valuable

feedback on their EA products For instance, one company showed many above

average scores except on the consistency quadrants (figure 4)

Figure 4: high scores except for the consistency quadrants Based on these findings it is expected that there is very little focus within the

company on being able to assess consequences, both intentional and unexpected,

from an EA product (recoverability and stability) Nor is there a focus on the ease

with which unwanted results can be changed (changeability) This is supported by

the fact that the product is not very clear on its own status (explicitness), leaving a

lot of room for unmature solutions to be interpreted as being ready for use in the

enterprise The EA products are coherent but now must focus on being consistent for

the enterprise

Figure 5 shows the overall score taking all 26 participating products into account

This benchmark graph shows 5 attributes that score lower than 50%, the lowest

being the completeness attribute As stated in the introduction of this paper and

proven with our developed instrument: “Dutch EA products are not complete but

they are attractive.”

Figure 5: benchmark EA products in the Netherlands

We conclude that EA product quality is measurable in a scientific way We hope that the instrument we developed will help the Enterprise Architecture industry improve itself and its image

Authors

Elise Veltman – van Reekum is working as a content architect at Kluwer B.V

where she uses her enthousiasm for architecture as a means to sharpen issues and

create sustainable solutions on the edges between Business, Content and IT

Marlies van Steenbergen is principal consultant enterprise architecture at Sogeti

Nederland B.V As such, she has guided many organizations in the implementation of

an effective enterprise architecture practice She is one of the founders of DYA and

co-author of “Dynamic Enterprise Architecture, How to make it work” and “Building

an Enterprise Architecture Practice” Marlies has published several articles and is a

frequent speaker at architecture seminars both at home and abroad

Martin van den Berg is an architecture service line manager at Sogeti Nederland

B.V In this role he is responsible for the development of architecture services and

expertise in Sogeti He is one of the founders of DYA and co-author of “Dynamic

Enterprise Architecture, How to make it work” and “Building an Enterprise

Architecture Practice” Martin van den Berg is also chairman of the architecture section of the Dutch Computer Society Martin has published several articles and is a much sought after speaker at architecture seminars

Dr Rik Bos is assistant professor Information Science at the department of

Information and Computing Sciences of the Utrecht University Among other areas

he takes part in the Master education on Enterprise Architecture He was involved in several projects that built IT under architecture

Prof dr Sjaak Brinkkemper is a professor of Information Science in the

department of Information and Computing Sciences of Utrecht University He leads a group of 15 researchers specialized in product software, in particular the

methodology of product development and implementation, and in entrepreneurship

in the product software industrie

He is a holds a Ph.D in Computer Science from Nijmegen University He has written six books and many articles on the area of information system development, method engineering and product software