Hands-On Microsoft SQL Server 2008 Integration Services part 57 ppt

Data Warehouse Design Approaches Now we know a bit about a data warehouse: that it keeps years of history, that it keeps data in granular as well as aggregated format, and that the prima

of an ODS cannot provide answers to such complex questions because either the data

is not available in the ODS systems or the data model doesn’t support such analysis This results into creation of a data warehouse or a decision support system A data warehouse or a DSS collects data from OLTP or ODS systems and might keep it

in multiple forms that is, in its most granular form and in aggregated form A data warehouse keeps data for longer periods of time (generally spread across several years) even after it has been deleted from the source systems

Data Warehouse Design Approaches

Now we know a bit about a data warehouse: that it keeps years of history, that it keeps data in granular as well as aggregated format, and that the primary function of a data warehouse is to do business intelligence or analytical reporting So the next question is how we design a data warehouse There are two primary schools of thought—top-down and bottom-up approaches—and both are good and relevant to their own applications but also have associated risks involved

Top-Down Design Approach

Bill Inmon, who is best known as the father of data warehousing, has defined this approach in which he suggests a data warehouse to be at the center of the Corporate Information Factory (CIF) designed using a normalized data model The Corporate Information Factory approach takes the holistic view of the enterprise and its informational needs In such a data warehouse, data is collected from most

of the organization’s operational systems and is held at the atomic level, that is,

at the lowest level of detail Further, the subject-oriented dimensional data marts containing aggregated data are built from the central atomic data warehouse to serve the departmental needs As the data warehouse becomes a source system for all the analytical data marts and organizational reporting, this creates a consistent view—a single version of truth across the enterprise It is easier to realign the data warehouse built with this approach to support business changes, and the data marts can be recreated easily from the central data warehouse However, the downside of this approach is that the difficulty involved in building such a data warehouse, collating the entire enterprise data from almost all the operational data sources, makes this a very large project So, implementing this project requires a high upfront investment, a large effort, and a long time to deliver This long delivery time results in users losing patience and developing their own solutions, thus deviating from the original point

of consolidation The departments or business units generally don’t buy into such projects with large delivery timescales

Bottom-Up Design Approach

In the absence of a central data warehouse, when departments or business units

need a data mart for a specific business process, they can’t wait for the central data

warehouse to be built but create their own departmental data mart Ralph Kimball,

who is a well-known author on data warehousing, is a proponent of this approach, for

which he suggests creating business-specific dimensional-modeled data marts first

These data marts can contain atomic as well as aggregated data in a denormalized data

model These data marts are created in phases and are linked together via conformed

dimensions A strict discipline is adopted to implement conformed dimensions that

are consistent with each other and use the same structure, attributes, and attribute

values Either these dimensions are exactly the same, including the keys, or one

is a perfect subset of the other The backbone of this approach is the conformed

dimension’s principle, and management of the conformed dimensions is fundamental

to maintaining integrity of the data warehouse Such a structure in which a data mart

is linked to another data mart via a conformed dimension is called data warehouse bus

architecture Later these data marts are joined together to create an enterprise data

warehouse Because dimensional data marts are at the center of the bus architecture

approach, the data model is simple to understand and use The dimensional or star

schema data model enables queries to retrieve data very efficiently Implementing a data warehouse using this approach realizes business investment quite quickly—as soon as

the data marts start functioning The data warehouse is developed in phases and hence

is more feasible than the central data warehouse approach The downsides of choosing

this approach are complicated data loading routines to maintain integrity within the

data marts and the difficulty in modifying data warehouse structures when business

changes occur

Data Warehouse Architecture

Based on the previous two design approaches, you can have two different architecture

layouts

Centralized EDW with Dependent Data Marts

This architecture is based on Bill Inmon’s Corporate Information Factory model

As the name implies, in this approach the data and applications reside on a central

mainframe system In this model, the data marts are fed exclusively from the centralized EDW in order to assure the single version of truth, thus creating a structure in which

the data marts depend on the centralized EDW Large enterprises use this approach, as

it provides some key benefits that are vital to their success However, there are instances

when this approach has failed to achieve desired results due to long deployment periods and huge upfront investments The benefits and the issues involved are listed here:

A single version of truth is the biggest reason why enterprises adopt this approach c

Centralized data management makes it easy to apply corporate standards and c

practices and allow businesses to comply with the legal requirements; a recent example of compliance is the application of PCI DSS standards issued by the PCI Security Standards Council

With a holistic approach, the development of such a warehouse does not follow c

the general principle for phased-iterative development

Due to their implementation plans, centralized EDWs can become quite inflexible c

When business units want to create analytical data marts, they get frustrated by this inflexibility and start moving away from this approach by creating federated data marts

Historically, the systems deployed in this architecture have been proprietary to a c

vendor that is slow to respond to fast improvements in technology and hence, they

do not benefit from the advances in general-purpose computer systems

This is a high-risk approach, as implementing a centralized EDW is very expensive c

with a huge upfront investment that is realized only after the data warehouse project is completed and data marts are beginning to emerge The maintenance and scalability costs have been found to be high in such an approach due to inflexibility All these costs add up to a very high total cost of ownership

Distributed Independent Data Marts

This architecture is based on the Ralph Kimball’s Bus Architecture approach and

is opposite to the monolithic design of a centralized EDW In this architecture, distributed departmental data marts are created as and when the need arises These data marts are designed independent of other data marts, and they get their data from operational source systems rather than centralized data warehouse These data marts are highly relevant to their departments or the business units and are quick to build,

as the scope is generally quite clear However, due to the lack of control over their implementation, they often result in many versions of truth and are very difficult to keep consistent across the enterprise The benefits and issues affecting this architecture are listed here:

The data marts are easy and relatively quick to build The cost to build one data c

mart seems low, as the return on investment starts pretty soon

Though the cost of developing a data mart seems low, the overall cost to develop a

c

distributed data warehouse is not low The cost is spread across several data marts

and remains unnoticed in the overall expenditure

The data management across independent data marts is not easy A lot of redundant

c

data exists across many data marts

Many versions of truth exist across an enterprise, and a data mart does not align

c

with another department’s data mart, though the dimensions may align

Data Warehouse Data Models

You can choose either of the two data modeling techniques–Entity-Relationship (ER)

modeling and dimensional modeling—depending on your application scenario The entity

relationship model has been in use much longer than the relatively new dimensional

model

Entity-Relationship Model

The ER modeling is more popular in operational applications or OLTP systems,

though it is also used in data warehousing as well This is a requirement in the

top-down approach, where a central data warehouse is built using the ER model There are

several graphical tools that help you to create an entity-relationship diagram (ERD) to

conceptualize data These tools primarily use three basic constructs: entity, relationship, and attribute

Entity

c An entity is a concept, real or abstract, about which information is

collected It represents a class of objects such as products, an object such as

a car, or an event such as a sale that can be classified by their properties and

characteristics It will usually have a business definition and is defined uniquely

using primary keys in the data model

Relationship

c A relationship is an association among entities in a model and

indicates how two or more entities are related to one another For example, a

customer owns a car—this indicates the relationship, how the customer is related

to the car It is represented by a line drawn between entities The entities are related

to each other in differing cardinality, which defines the maximum number of

instances of one entity related to a single instance of another entity The relationship

has one-to-one, one-to-many, and many-to-many cardinalities

Attributes

c Both entities and relationships have attributes that describe their

characteristics or properties For example, a car has a VIN number attribute

The relational databases are designed using the normalized ER model to remove redundant data Six normal forms have been defined to date, but a database is considered adequately normalized if it is in third normal form (3NF) Normalization

is a systematic process for assigning attributes to entities to ensure that a database is integral by breaking down information to its smallest divisible parts and removing data duplication It is an incremental process, which means that to be transformed into 3NF, the entities must first qualify for 2NF The 1NF removes repetition in data by creating one-to-many relationships between master and detail entities For example, you will remove repeating similar columns from a table into another table 2NF takes removal of repeating data a further step by removing duplicate rows of data from a table into a separate table 3NF removes the columns that are not dependent on the primary key and resolves many-to-many relationships into unique values

As the normalization level increases, the data is further broken down and granularity

of data is increased Such highly granular data models are very efficient in returning small amounts of information or updating small amounts of data This is required by OLTP systems that have many users working on small pieces of information That’s the reason the ER Model is highly successful in relational database applications However, the requirements of a data warehouse are different The queries are usually small in number, but they can perform huge I/O activity on the server These requirements are met with the dimensional model

Dimensional Model

Dimensional modeling is a relatively newer modeling technique than ER modeling Recent trends in modeling preference favor dimensional data modeling, as it is simple

to build, is easily to understand even by business folks, and aligns with the questions usually asked of a data warehouse This model is very efficient at summarizing values and presenting data to analytical tools This model keeps the numerical values called

facts in one table, while the attributes that measure these values are grouped together in

tables called dimensions This structure makes it particularly suitable to answer business

questions such as sales this quarter or the sales this quarter compared to sales the previous quarter

Fact

The numeric values along with some contextual data are stored in fact tables A data

warehouse contains one or more subject-oriented fact tables A fact typically represents

an item, an event, or a business transaction such as sale of an item that can be used to perform business analysis Further, a fact consists of some columns containing values and some foreign keys linking to dimensions As the transactions are added to the fact

table, it can soon become quite large; consider that each transaction could represent a

row in the fact table

It is the granularity that can really affect the size of your fact data You have to

understand the business requirements completely—what they call for now and in

the future—to decide the level of detail you want to keep in the fact data You might

choose to keep every transaction, or you might choose to keep aggregated data at the

day level if business is never going to drill down to the transaction-level detail Thus

the lower the granularity level, the more the data and hence, the more disk space will

be required However, disk space should not be an issue, because data warehouses are

meant to be large in size Always be careful while choosing granularity, as a change in

granularity means the whole data warehouse has to be reseeded

Measure

A measure is closely related to a fact, and sometimes the terms are used interchangeably

A measure is what you want to measure, and the fact is a measure with context So, a

measure is a numeric value used to indicate the performance of the business, and the fact

equates to a row in the Fact table A measure is used in combination with dimension

members, while the value is taken from facts For example, TotalSales-by-year and

SupplyCost-by-month are measures

Dimension

A dimension contains the same type of information broken down in levels of interest For

example, a time dimension can contain year, quarter, month, and day levels A dimension

contains the information that a business wants to analyze the facts with This information does not change very often Typically, a dimension table is relatively quite small compared

to a fact table A data warehouse can have many dimensions attached to each fact table

Some of the common dimensions could be Time, Product, Employee, Location, and

Customer A dimension is made up of members and hierarchies

Dimension Members

Member of a dimension are arranged in levels, for instance, members in a time dimension have different levels: day, week, month, quarter, and year Similarly all cities, states, and

countries are members of a geography dimension While analyzing the data, you can choose

any dimension member level and associated facts will be used in analysis automatically

Dimension Hierarchies

The members of a dimension can be arranged in a hierarchical order with multiple

levels to create dimension hierarchies You can create more than one hierarchy and a

dimension member can participate in more than one hierarchy For example, you can

create two hierarchies for time dimension—one with Day, Month, Quarter, and Year

as levels and the other one with Day and Week as levels

Dimension Types

Dimensions can be designed in different ways to meet specific business functions and

to enhance performance Four types of configurations are covered here

Conformed Dimension

At times you will use a dimension in more than one subject-oriented data marts If you are keeping the dimensions exactly same or are sourcing them from the central data

warehouse, obviously making them the same, then these dimensions are called conformed

dimensions A conformed dimension does not need to be exactly the same as the main dimension; it is still conformed to the main dimension if it is a subset of the detailed dimension In this case, the attributes in a conformed dimension need to be labeled exactly

in the same way as in a detailed dimension The most common example could be a date dimension used across many data marts having same attributes such as date, month, quarter, and year

Junk Dimension

The business data generally contains some attributes that are not related to any dimension but are associated more with the fact These can be easily identified, as generally these attributes represent themselves in the form of indicators or flags For example, in a car rental business flags such as IsDamaged, IsStolen, and IsExchanged are common in the databases These flags are not exactly part of any dimension, but businesses do want to analyze data using these flags If you leave them with the fact data, the performance of the queries will be very poor due to the large size of the fact table, and indexes won’t help due to yes/no nature of such flags You could put them in their own dimension, in which case you would have as many dimensions as there are such attributes Very frequently you will see that the number of indicators or flags that exist in the data reaches 20 plus In this case, your data mart design will be cluttered with lots of dimensions that have only one member, enough to confuse users

A recommended approach is to club all these nonrelated attributes and put them in a

table, thus creating a junk dimension For instance, you can select distinct combinations

of the attributes and add them to the junk dimension where each distinct row is assigned a surrogate key that is referenced in the Fact table Keeping the flags and indicators in one dimension makes it easier for users to find out these attributes and is useful in that the queries perform much better

Degenerate Dimension

Another type of attribute in the data is the number associated with each business

transaction, such as an invoice number or a ticket number Though these attributes are

not actively used in analysis, every now and then businesses do have a requirement to

look for such attributes, as they link back with operational systems These attributes

are also very useful in reconciling the data warehouse with ODS systems When the

grain of the fact table is the same as that of a transaction, the likelihood of a degenerate

dimension increases As the fact grain is the same as that of the degenerate dimension,

so sometimes it forms an integral part of the primary key of the fact table A degenerate dimension should exist along with the facts in the fact table There is no point in

creating a separate dimension table for a degenerate dimension, as it will grow with the

fact and will become quite large

Role-Playing Dimensions

These types of dimensions exist when the same attribute is used multiple times For

instance, a fact row could contain SaleDate and DeliveryDate columns, both pointing

to the date attribute of the date dimension In this case, the key of the date attribute

of the date dimension is used multiple times in the fact row and is often referred to as

a role-playing dimension You create a table alias or a view to use the date dimension

foreign key in the fact table

Loading a Dimension Using

a Slowly Changing Dimension

Loading a static dimension is a simple one-off task, but you will come across dimensions

that change with time and will find loading such a dimension a challenging task Some of

the data warehouse dimensions do not change that often For instance, a date dimension’s members stay as is and never change By contrast, other business dimensions do change

with time as a business evolves; for example, a product could change in size and volume

Typically, a row in the dimension table will have different attribute requirements:

Some attributes never change, such as IDs, or in the case of a car, a VIN number

c

Some attributes will change but a business always want to see the current value;

c

for instance, a business may not bother to see the old description of the product

if the description of a product is changed This type of change is called a Type 1

change in which the attribute value is overwritten and a business cannot go back

and see the old value or learn when the change in value has been applied

Some attributes will change and the business will want to see the current value as c

well as the historical value This happens when a business is interested in analyzing the facts before and after a particular change has been applied For instance, a business might want to see the effect on sales when the size of a can of beans is changed from 400 g to 350 g This type of change is called a Type 2 change Most of the business requirements can be covered with use of the preceding types c

Other types of changes have been defined such as Type 3 and so on; however, their primary function is to improve storage efficiency and query performance These change types are not covered here Refer to data warehousing books if you want to know more about them Also, the SCD transformation in SSIS supports only up

to Type 2 changes

You have studied the Slowly Changing Dimension (SCD) transformation in Chapter 10 and have done a Hands-On exercise as well Here just to recap: the SCD transformation is designed to help you load a dimension that changes in time, which

is generally a challenge with an ETL tool The SCD transformation supports the following attribute change types to support the previously mentioned requirements

Fixed Attribute Change Type

c This change type supports the attributes that do not change (Type 0) and align with the first scenario mentioned previously

Changing Attribute Change Type

c Using this change type, you can load the attribute values that are Type 1 changes, and it overwrites the existing values with the new values This is an in-place modification

Historic Attribute Change Type

c In this change type, a new row is added that will be valid for the current or future transactions Typically, three columns are used to handle this type of change—StartDate, EndDate, and IsActive When

a row is getting a Type 2 update, it will update the IsActive flag to ‘N’ and will timestamp the EndDate with the current date and time to indicate that this row is no longer active, while the activity period of this row can be found using StartDate and EndDate Also, at the same time, a new row is added with same values in all the columns except in the Type 2 column that is getting the update

In this Type 2 column, the updated new value is inserted, StartDate gets the current date and time stamp, EndDate is kept as null, and the IsActive column gets a ‘Y’ value to indicate that this row is active for the particular member

Among other methods, SCD transformation can be tested to load a dimension If you think that your dimension is too huge and the SCD transformation is not a fit for the purpose, you can always create a script in your package to load a slowly changing dimension

Data Model Schema

You can model your data model with two different types of schema

Star Schema

As mentioned earlier, a data mart is a subject-oriented mini–data warehouse and will

have one or few fact tables surrounded by a relatively large number of dimensions

When these are drawn on a piece of paper, the structure looks like a star, hence the

term star schema The star schema is a simple model in which the denormalized

dimensions are connected to facts using foreign key relationships This model has

become a building block in dimensional modeling In a large data warehouse where

multiple fact entities can exist, you can imagine a multiple star schema model with each

dimension connected with several fact entities Figure 12-1 shows a simple example of

a star schema data model

FactSales

DimCustomer

DimSalesTerritory

DimDate

DimSalesPerson

Figure 12-1 A simple star schema data model