Hands-On Microsoft SQL Server 2008 Integration Services part 20 ppt

Message Queue Task Microsoft Message Queuing MSMQ is a Windows service that allows applications or SSIS packages to send and receive messages using a queue.. You can choose from one of t

If you have chosen ADO or ADO.NET Connection Manager in the ConnectionType field earlier, the IsQueryStoredProcedure field becomes available for use and you will be able to specify that the SQL statement to be run is a stored procedure

When you select OLE DB Connection Manager in the ConnectionType field, you can specify in the BypassPrepare field whether the task should skip preparing the query step before running it When SQL queries are run, the SQL statement is parsed and compiled in the first step and then executed based on the execution plan prepared earlier in the compilation step By default the BypassPrepare field is set to True, which means when the query is run the first time, it parses and compiles the SQL statement

in the first step and keeps the execution plan in the cache of the database engine and

in the next step uses this execution plan to execute the query So when the query is run the next time, it doesn’t parse and compile the statement but rather uses the existing execution plan, saving time required to parse and compile If you’re running a query multiple times, it is faster and efficient if you use a prepared execution—however, if you’re running the query only once, it is not a recommended option

Parameter Mapping Page

If you are running a parameterized SQL statement, you can map variables to the parameters

in this page The interface on this page is relatively simple and provides self-explanatory fields and options You can click Add to add a parameter mapping and then click

in each field to select the available values from the drop-down lists; however, in the Parameter Name field, you have to type in a value The configuration of this page has been covered in the “Using System Variables to Create Custom Logs” Hands-On exercise in Chapter 3

Result Set Page

If you are using a query that will return a result set and you have selected a value other than None in the ResultSet field on the General page, you can use this page to map the row set to a variable, which can be used later in the package

Expressions Page

Using this page, you can build a property expression using SSIS expression language

to update the property dynamically at run time In the Expression Builder, you can use System or User variables, functions, type casts, and operators exposed by expression language to build an expression

Bulk Insert Task

When you get large amounts of data in the flat files, say from the mainframe systems

or from third parties, and you want to import this data from the flat files into an SQL

Server, you can use the Bulk Insert task This task is the quickest way to copy large

amounts of data into an SQL Server table or view

Note the following when deciding whether to use this task:

The Bulk Insert task does not perform any validation or transformation on the

c

data while importing data from the source text file to the SQL Server database

table or view So, it is more suitable in cases where the data is coming from a

reliable source or you intend to apply business processes in later stages A similar

component—SQL Server Destination—is provided in the data flow task that

can provide the performance of a bulk insert You will be using SQL Server

Destination if you want to perform transformations on data before loading

and your target server is the local server SQL Server Destination is covered in

Chapter 9 in detail

When you embed this task in an SSIS package Control Flow, only the members

c

of the sysadmin fixed server role can run the package

You can use an XML- or non-XML-formatted file in the Bulk Insert task

c

A formatted file is used to store format information for each field in a data file

in relationship to a specific table and provides all the format information that

is required to bulk-import data This format file must be located on the server

executing the SSIS package containing the Bulk Insert task

During the import process, some of the rows may fail to import and in turn can fail c

the task You can still run this task successfully by increasing the allowed number of errors by specifying a value in the MaxErrors option However, the rows that fail to

import cannot be logged or extracted out If you want to capture the failing rows,

you need to consider alternative ways of importing data using the Data Flow task

and capture the failing rows in the exception files by way of error outputs

Message Queue Task

Microsoft Message Queuing (MSMQ) is a Windows service that allows applications or

SSIS packages to send and receive messages using a queue Message queuing is available

in Windows 2000 and later released as a standard operating system component that

has the advantages of Active Directory Integration This service can be extended using

the Message Queuing Connector to heterogeneous data sources such as CICS (IBM’s

Customer Information Control System) or UNIX

In the SSIS world, message queuing means one package will be sending a message

to a queue while another package will be receiving that message from the queue These packages can be on different servers and may not be running simultaneously Using MSMQ, you can reliably exchange data between packages that may not be running on the same server and might be separated in time throughout your enterprise Following are some of the scenarios in which you may be using the Message Queue task:

Send a message from an executing package to the package that is waiting for it to c

complete so that it can start running

If you have a small window of time to finish a large workload and you decide to c

distribute your workload across many servers to utilize the processing power, you can coordinate the operations on different servers using message queuing between packages

SSIS packages can communicate with the applications that utilize message c

queuing

Send output from a processing package to a waiting package on the other computer, c

where the data enclosed in the message will be processed further

You can configure this task into either send mode or receive mode Select Send Message in the Message field to configure it into send mode or select Receive Message

in the Message field to configure it into receive mode The package that is running the message queue task in the receive mode can be configured to either run and wait for the message in the queue using a looping construct or run on a schedule that is later than when the message is dropped in the queue just to pick up the message SSIS is more suitable for the applications that can be satisfied with the latter implementation If you need to wait for messages, for example, your application is sending data in message format as the transactions happen, Biztalk is more suitable for such implementations Biztalk server is designed to support such requirements and puts negligible load on the server if it has to wait for messages, whereas SSIS will consume resources, as it has

to keep looping and stay in memory The message queue task is provided in SSIS to transfer the information that is relevant for the remaining ETL to complete rather than read or load data from a transactional system

You can choose from one of the different types of messages to send or receive using the Message Queue task: Data file message, String message, or Variable message When the task is configured in the receiving mode, you also get an additional String message to variable MessageType choice As in other Integration Services tasks, the

available fields in this task are dynamic and change depending upon your choice to send

or receive a message or on message type

Data file message

c Used to send or receive a file that contains the message To

send a data file message, you specify Data File Message in the MessageType field

and specify the path of the file in the DataFileMessage field on the Send page

of the Message Queue Task Editor When receiving this type of messages, you

select Data File Message in the MessageType field and specify the name of the

file in the SaveFileAs field into which you want to save the message Also, you can

choose to overwrite the existing file, and optionally apply a filter to receive the

message only from the specified package You specify the full path and package

name in the Identifier field on the Receive page

Variable message

c Used to send or receive one or more variables To send variables,

you specify Variable Message in the MessageType field and one or more variables

in the VariableMessage field on the Send page of the Message Queue Task Editor While receiving the variables, you specify Variable Message in the MessageType

field, specify the name of the variable in the Variable field to receive the message

into, and optionally choose to apply a filter to receive the message only from the

package specified in the Identifier field on the Receive page

String

c message Used to send a text string To send a text string, you specify String

message in the MessageType field and type a text string in the StringMessage field

on the Send page of the Message Queue Task Editor While receiving the text

string, you specify String message in the MessageType field and optionally specify

to compare the incoming string in the Compare field with a user-defined string

specified in the CompareString field on the Receive page The string comparison

options in the Compare field can be Exact Match for an exact comparison, Ignore

Case for case-insensitive comparison, or Containing for a substring match

String message to variable

c Used to pass the source message that has been sent

as a string to a destination variable and is available only when receiving messages

To configure a text string message to be passed to a variable, you specify String

Message To Variable in the MessageType field, specify the name of the variable

in the Variable field to receive the text string into, and optionally specify to

compare the incoming string in the Compare field with a user-defined string

specified in the CompareString field on the Receive page The string comparison

options in the Compare field can be Exact Match for exact comparison, Ignore

Case for case-insensitive comparison, or Containing for a substring match

To use the Message Queue task to send and receive messages, you first need to install the Message Queuing service and then create messaging queues You can create either a public or a private queue, depending upon whether you have installed the Active Directory Integration component of the Message Queuing service A public queue is created in an Active Directory environment to publish its properties and description to the Active Directory A private queue does not publish itself to the Active Directory but works on the local computer that holds the queue

The Message Queuing service has the following components available in Windows

2008 Server for installation:

Message Queuing Server

Queuing functions such as guaranteed message delivery, efficient routing, improved security, support for sending messages within transactions, and priority-based messaging It can be used to implement both synchronous and asynchronous messaging solutions

Directory Service Integration

whenever the computer belongs to a Windows domain (Windows 2000 and later) With this component, you can publish queue properties to the directory, authenticate and encrypt messages using digital certificates registered in the directory, and route messages across directory sites Public queues are configured and used when using this component

Message Queuing Triggers

with triggering functionality of a COM component or a standalone application, depending upon the filters that you define for the incoming messages in a given queue

HTTP Support

c Enables sending or receiving of messages over HTTP transport with proxy settings configured using the proxycfg.exe tool

Multicasting support

c Enables queuing and sending of multicast messages to

a multicast IP address

Routing Service

c Available in Windows 2008 Server to provide routing of messages using the store and forward mechanism in Active Directory transport between different sites or within the same site

Windows 2000 Client Support

support for Message Queuing clients on Windows 2000 computers in the domain

Message Queuing DCOM Proxy

of a remote Message Queuing Server

For proper functioning of the Message Queue task, make sure you have installed the

Message Queuing Server and the SQL Server Integration Services service When you

install SQL Server 2008 without specifically selecting the Integration Services service,

you may still be able to use BIDS to design and run Integration Services packages

However, as this is a partial installation of SSIS, not all tasks will run properly, and

the Message Queue task is one of those that won’t For the Message Queue task to be

functioning properly, you must install Integration Services fully during installation of

SQL Server 2008

Next, you will import the expanded files into the Campaign database in SQL Server

using the Message Queue task and the Bulk Insert task in the following Hands-On

exercises

Hands-On: Importing Expanded Files

Dealers of Adventure Works Bikes submit the sales reports to an FTP server, and they need

to be downloaded from this server and imported into the SQL Server database You have

already downloaded, expanded, and archived these report files in the previous exercises,

and now you have DealerSales01.txt and DealerSales02.txt files in the Downloads

folder ready to be imported to the Campaign database However, you want to run this

new package independent of earlier packages, and this may occur at a different time

during the day

Method

We can meet this objective in several ways: We can use the Bulk Insert task to import

the data to our existing table—but we have multiple files, so we can use the Foreach

Loop Container to read the filenames one by one and pass them to the Bulk Insert

task to import multiple files one at a time However, as we want to keep the packages

independent from one another and want to run the second package only when the

first has completed successfully, the use of messages from the first package to pass the

information to the other will be a better solution Though in this case you will not be

following the shortest and easiest solution option to achieve your goal, the solution

you are going to follow is quite interesting, and I’m sure it will be relevant in real-life

scenarios

You will add the Message Queue task at the end of the Archiving Downloaded

Files package to send filenames in the messages, which will be read by the Importing

Expanded Files package using the Message Queue task in the receiving mode As

the package won’t have to read filenames from the file system, you will not be using

the Foreach Loop Container; instead, you will use the For Loop Container to read multiple messages one by one Here’s the step-by-step method:

1 Install the Message Queuing service and create message queues

2 To keep it simple and applicable to most of the users, for the sake of this exercise I’ve used a Windows Server 2008 machine with a private queue only If you’ve a different environment, refer to Microsoft SQL Server 2008 Books Online for more details to create a message queuing environment suitable to your requirements

3 Configure the Archiving Downloaded Files package to send filenames In this step, we will add Message Queue task in the already created Archiving Downloaded Files package

4 Build the Importing Expanded Files package And the final step is to execute this package and check that the two messages have been picked up from the queue and the text files have been imported into the SQL Server Campaign database

Exercise (Install Message Queuing Service)

Let us start this exercise by installing the Message Queuing service You will also create message queues while learning about private and public queues

1 From the Server Manager, click the Add Features link to start the Add Features Wizard

2 In the Select Features dialog box, expand the Message Queuing group and select the Message Queuing Server from the list Click Next to go to Confirm Installation Selections page

3 Verify the selection and click Install to install the Message Queuing Server component

4 When the installation is completed successfully, click the Close button in the Results page to close the wizard

5 Again in the Server Manager, expand the Features node in the left pane of the window to see the installed Message Queuing feature

6 Right-click the Private Queues folder and choose New | Private Queue from the context menu This will open the New Private Queue window In the Queue

name field, type SSISprivQ and then click OK.

Note that another difference between a private queue and a public one is that

the private$ is attached to the full name of a private queue The New Private Queue window shows that the queue will be created in your computer name with private$ added in front of the name you type in So, the full path of the queue will be ComputerName\private$\SSISprivQ If this were the public queue in which we specify the queue name as SSISpubQ, the full path would have been ComputerName\SSISpubQ.

7 Expand the Private Queues folder under the Message Queuing from the left pane

of the Computer Management window Further expand the SSISPrivQ folder

and you will see a Queue messages subfolder This is where the Message Queue

task will be delivering the messages and the messages will wait for a receiving

Message Queue task to pick them up

Exercise (Configure Archiving Downloaded

Files Package to Send Filenames)

Remember that in the Archiving Downloaded Files package, the Foreach Loop

Container reads the filenames of zipped files lying in the C:\SSIS\downloads folder

and then passes those names to the File System task, which moves the zipped files to

the Archive subfolder To accomplish this, the package uses a variable named fname

to pass the zipped files names Because you want to import text files, not the zipped

files, you will derive a new variable called txtfname to contain the text files names from

fname variable This new variable will then be sent through the Message Queue task as

a variable message

8 Run BIDS and open the Control Flow Tasks project In the Solution Explorer

window, double-click the Archiving downloaded files.dtsx package to open it

9 Open the Variables window Click the Auto Hide pushpin to dock the Variables

window on the left side of the screen On the Control Flow surface, click the

Enumerating Files Foreach Loop Container and switch to the Variables window

Add a new variable named txtfname in the Enumerating Files Scope and set Data

Type to string Press f4 to open the Properties window The Properties window

will open up by default on the right side of the screen, showing properties of the

txtfname variable (Sometimes you may not see properties for the object in which

you are interested—this is because of your selection of the items on the designer

If this happens, make sure you’ve selected the Enumerating Files Container and

then clicked the txtfname variable to see the properties for this variable.)

Scroll through the properties and locate the EvaluateAsExpression property This

property allows you to enable the variable to use the results of an expression as its

value That is, you can write an expression, which will be evaluated at run time

and the result of this evaluation will be used as the value of the variable

Select True for the EvaluateAsExpression property Type the following text in

the Expression property:

SUBSTRING(@[User::fname] , 1, LEN(@[User::fname]) -3) + "txt"

This expression will evaluate txtfname as C:\SSIS\downloads\DealerSales01.txt

from the fname variable when it equals C:\SSIS\downloads\DealerSales01.zip If

you look further down in the properties window in the Value field, you will notice

(see Figure 5-9) that the full filename in fact has been resolved as txt You can see that nothing appears in the name portion of the file path name because the

fname variable has not yet been populated (by the Foreach Loop Container) in the

design mode However, at run time, the Foreach Loop Container provides a value

for the fname variable and the name portion of the filename gets populated in the

file path name

10. Drag the Message Queue task from the Toolbox and drop it in the Enumerating Files (Foreach Loop) Container below the Archive Downloaded Files (File System) task Drag and drop the green arrow, the precedence constraint, from below the Archive Downloaded Files task on to the Message Queue task The package will look as shown in Figure 5-10 after you’ve configured the Message Queue task

11. Right-click the Message Queue task and choose Edit from the context menu Type the following on the General Page of the task editor:

Name Send variable message

Description This task sends the txtfname variable to the Importing expanded files

Figure 5-9 Deriving the txtfname variable using property expressions

If you want to send this message in the format that is acceptable to SQL Server

2000 Message Queue task, you can use the Use2000Format field For now, leave

the default False value set in this field

12. In the MSMQConnection field, select to create a new connection In the

MSMQ Connection Manager Editor, type the following:

Description Connection Manager for private Queue SSISprivQ

Path YourComputerName\private$\SSISprivQ

You can use a dot (.) instead of YourComputerName to indicate your local computer.

Figure 5-10 Archiving the downloaded files package after adding Message Queue task