Hands-On Microsoft SQL Server 2008 Integration Services part 21 pdf

You may want to test the connection by clicking Test; once you get the “Test connection succeeded” message, click OK twice to return to the Message Queue Task Editor.. The mincounter and

You may want to test the connection by clicking Test; once you get the “Test connection succeeded” message, click OK twice to return to the Message Queue Task Editor

13. Leave the Send Message option selected in the Message field The settings in General page should be like the ones shown in Figure 5-11

14. Go to the Send page Leave UseEncryption set to False This field can be used to encrypt your messages using an encryption algorithm (RC2 or RC4) specified in the EncryptionAlgorithm field

Figure 5-11 Message Queue task configured to send messages

15. Select Variable message in the MessageType field, as you will be sending a

variable value in the message Select User::txtfname in the VariableMessage field

Click OK to close the editor window

16. Save the package by pressing ctrl-shift-s

17. Using Windows Explorer, move the DealerSales01.zip and DealerSales02.zip

files from the C:\SSIS\downloads\Archive folder to the C:\SSIS\downloads

folder that have been moved to the Archive folder in earlier exercise

18. Switch to BIDS and press f5 on the keyboard to start debugging When the

package runs, notice that the tasks change color from yellow to green twice,

indicating that the package has moved two files and sent out two messages When all the tasks turn green, stop debugging from the Debug menu or by pressing

shift-f5 Close the project by choosing File | Close Project

19. Switch to the Server Manager and notice two messages in the Queue messages

folder under SSISprivQ (You might have to refresh the folder if the window was

already open.)

Exercise (Build Importing Expanded Files Package)

In this part of the exercise, you will build a new package, which will pick up variable

messages sent by the Archiving Downloaded Files package earlier from the SSISprivQ

messaging queue

20 In the Solution Explorer, add a new package and rename it Importing expanded

files.dtsx.

21. Start building the package by dropping the For Loop Container on the Control

Flow Designer surface

22. Right-click anywhere on the Designer surface and choose Variables from the

context menu Create three variables: filename, maxcounter, and mincounter with

the details shown in the Figure 5-12

Figure 5-12 Creating variables for the text filenames and For Loop Container

Be sure to specify a value of C:\\SSIS\\RawFiles\\DealerSales_placeholder.txt to

the filename variable and a value of 2 to the maxcounter variable The mincounter and maxcounter variables will be used in the For Loop Container, while the filename variable will be used to receive the variable value from the messages sent

by Archiving Downloaded Files package using the Message Queue task

23. Double-click the For Loop Container icon to open the For Loop Editor Type the following in the General page of the editor:

Description This For Loop Container imports DealerSales files one by one

InitExpression @mincounter = 0

EvalExpression @mincounter < @maxcounter

AssignExpression @mincounter = @mincounter + 1

As the initial value assigned to the mincounter variable in the InitExpression field

is 0 and the maximum value specified for maxcounter variable while creating it is 2,

the Import Loop will loop twice at run time, after which the expression defined in EvalExpression field will become false and the loop will stop running Click OK

to close the For Loop Editor window

24. From the Toolbox, drag and drop the Message Queue task within the Import Loop (For Loop) Container Open the Message Queue Task Editor window by double-clicking its icon and type the following on the General page:

Name Receive variable message

Description This task receives variable messages sent by ‘Archiving downloaded files’ package

25. Leave the Use2000Format field set to the default False value

26. Create an MSMQ connection manager in the MSMQConnection field exactly as specified in Step 3 of the preceding series of steps

27. Select Receive Message in the Message field Note that Send changes to Receive

on the left pane of the window Go to the Receive page

28. On the Receive page, select True in the RemoveFromMessageQueue field This value lets you delete the message from the queue after the Message Queue task has received it You may prefer to keep the message in the queue in situations when you have multiple subpackages to read a message

29. You can choose to display an error message if the task fails with timeout in the ErrorIfMessageTimeout field For now, leave it set to False However, if you select True in this field, you will be able to specify a time-out value in seconds in the TimeoutAfter field

30. Choose Variable Message in the MessageType field and No Filter in the Filter

field If you select the From Package option in the Filter field, you can specify the

package in the Identifier field

31. Specify User::filename in the Variable field, as shown in Figure 5-13

32. Click OK to close the Message Queue Task Editor

33. Drag and drop the Bulk Insert task from the Toolbox in the Import Loop

Container below the Receive variable message task Connect the Receive variable

message task to the Bulk Insert task using the on success (green colored line)

precedence constraint

Figure 5-13 Configuring the Message Queue task in receive mode

34. Open the Bulk Insert Task Editor and type the following in the General page:

Name Importing DealerSales

Description This task imports DealerSales files into Campaign DealerSales table

35. Go to the Connection page, click in the Connection field under the Destination Connection group of options, and choose <New Connection…> to open the Configure OLE DB Connection Manager window Choose an OLE DB connection manager from the Data Connections list to connect to Campaign database, which you created in an earlier Hands-On exercise The good thing about the OLE DB Connection Managers is that they are available for reuse

in other packages, as you have seen here If you skipped the earlier Hands-On exercises and do not see an OLE DB Connection Manager in this window, you will have to create a new connection manager by clicking New For details on how to create an OLE DB Connection Manager, refer back to the “Contacting Opportunities” Hands-On exercise in Chapter 4

36. Click in the DestinationTable field and then click the drop-down arrow to see the list of tables in the Campaign database OLE DB Connection Manager provides this list by establishing a connection to the database using the settings specified in the connection manager Select the [Campaign] [dbo].[DealerSales] Table from the list

37. You can choose to specify the format of the file to be imported either directly in the task or by using a file If you choose Use File in the Format field, you have to specify the name and location of the file in the FormatFile field, which appears on selection of the Use File option For this exercise, choose Specify in the Format field Leave the RowDelimiter field set to {CR}{LF} and choose Vertical Bar {|} from the drop-down list in the ColumnDelimiter field

38. Under Source Connection options group, choose <New Connection…> to specify

a File Connection Manager As you will be using filenames provided by the Receive Variable Message task in the form of variables, you will specify a filename here as a placeholder Specify C:\SSIS\RawFiles\DealerSales_ placeholder.txt in the File field of the File Connection Manager Editor (see Figure 5-14) You can create a blank DealerSales_ placeholder.txt file in case you’re having difficulty specifying it in the File Connection Manager

39. Go to the Options page, where you can specify the code page of the text file in the CodePage field Leave it set to the default RAW

40. When the data is coming from various database systems, the data type may be

in different formats The Bulk Insert task provides four default data formats that can be imported You can specify the data type of the input file using the DataFileType field The options are Character, Unicode (wide) Character, Native, and Unicode (wide) Native formats Leave it set to char

41. You can specify the number of rows in a batch in the BatchSize field The

rows specified in a batch are treated together and copied to the server as one

transaction The Bulk Insert task copies files by committing all the rows in

the batch as one transaction and moving over to the next batch for another

transaction In case of an error and failure of the task, all the rows in the batch

will be rolled back For example, if you set the BatchSize equal to 10,000 rows

for a table of 50,000 rows and the task fails at row number 25,001, the task will

fail with 20,000 rows inserted to the table A default value of 0 implies that

all the rows will be treated as a single batch—i.e., fail or commit totally in one

transaction

Figure 5-14 Configuring the Bulk Insert task to import text files

42. In the LastRow field, you can specify the value of last row at which the task should stop inserting data to the table Leave it set at the default value of 0 that means all the rows from the file will be inserted to the specified SQL table

43. The FirstRow field is quite useful in situations where you have a large amount

of data to import and the quality of data results in failing the process in between

By specifying where to start inserting the rows, you can avoid re-importing the rows that have already been imported Using this option in conjunction with the BatchSize option helps in achieving high levels of input performance with less rework, even though the data quality may not be good Leave the option selected

to the default value of 1

44. Moving to the Options section, you can choose any of the five options You can also select more than one option here, which then will be listed in a comma-delimited list

Check Constraints

c Checks the table and column constraints

Keep Nulls

c Imports blank columns from the text file as Null values

Enable Identity Insert

c Inserts explicit values into the identity column of the table

Table Lock

c Locks the table during import process

Fire Triggers

c Fires any existing triggers on the table while importing the data

Though none of these options will be of much help in this case, you can select Keep Nulls for this exercise

45. You can specify the names of columns on which to sort the data in the SortedData field This is effectively the ORDER BY clause in the bulk insert SQL statement Leave it blank, which means do not sort on any column

46. The rows that cannot be imported by Bulk Insert task are counted as errors You can specify a maximum number of errors—i.e., the number of rows to fail before the task fails—in the MaxErrors field Click OK to close the task

You have configured all the tasks and options within the tasks apart from specifying which files to import You can do this by mapping the connection string of the placeholder File Connection Manager to the value of the variable received by the Receive variable message task Let’s see how to do this

47. Right-click the DealerSales_placeholder.txt File Connection Manager and choose Properties from the context menu In the Property window, click in the Expressions field and then click the ellipsis button on this field In the Property Expressions Editor, click in the field below the Property column and select ConnectionString from the drop-down list Then click the ellipsis button under the Expression field to open the Expression Builder

48. In the Expression Builder, drag User::filename from the Variables list to

the Expression field, as shown in Figure 5-15 Click OK twice to finish the

configurations

49. As a last step before you run the package, make sure that you have two messages

queued, which were sent by the Archiving Downloaded Files package, in the

message queue folder in SSISprivQ Also, check that the DealerSales table in the

Campaign database does not have any record

50. In a real-life scenario, you will be running such a package under a schedule

that wakes up after the message has been dropped in the queue just to read the

message For our exercise, press f5 to run the package

Notice that the Import Loop task turns yellow, followed by the Receive Variable

Message task turning yellow and then green, indicating that it has successfully

received the first variable message Then Importing DealerSales turns yellow

and stays yellow for some time before turning to green, indicating that it has

successfully imported DealerSales01.txt file (Figure 5-16) This completes the

first iteration of the Import Loop task and the process repeats itself for the second iteration, after which all the tasks turn green and stop processing

Figure 5-15 Using Expression Builder to create a property expression for dynamically altering the connection string of the

file connection manager

To close the loop, check the Queue messages folder under SSISprivQ by switching over to the Server Manager and notice that the messages have been deleted Go

to SQL Server Management Studio and run the following query against the DealerSales table to see the imported data and total number of records:

SELECT * FROM DealerSales

You should see 242,634 rows displayed

51. Press ctrl-shift-s to save all the files in this solution and then choose File | Close Project

Review

In this exercise, you used the Message Queue task to send and receive variables from one package to another using the Windows Message Queuing service This can be quite useful for enterprise-wide implementations in which data and servers are scattered

Figure 5-16 Executing the Importing expanded files package

all over the network, and it makes sense to use all the processing power underutilized

in those servers to help you complete the nightly processes within the allocated time

You also used the Bulk Insert task to import files to an SQL Server table This is

the fastest method of importing data to an SQL Server table In this exercise, you

had encountered the For Loop Container and used it to iterate twice to receive two

messages using the Message Queue task and then import two text files into the SQL

Server table using Bulk Insert task Last but not the least, and perhaps the most

important thing you’ve learned in this exercise, you used property expressions to change

ConnectionStrings of the File Connection Managers dynamically at run time with the

help of variables

Execute Package Task

The Execute Package task brings the benefits of modular design to SSIS packages It

allows an SSIS (parent) package to run the other (child) packages as part of a workflow This task can be used to run packages stored either in the SQL Server MSDB database

or on the file system

While developing a solution for a business problem, you tend to build smaller

packages meeting specific needs This approach is recommended, as it helps your

solution to be modular, which is easier to debug This also helps you achieve quick

results on some of the specific requirements Once you have built all the modules of

your complete solution, you can use the Execute Package task to fit all the smaller

packages together to form an enterprise-wide solution

Another benefit of following a modular design for SSIS packages is less work when

you have to modify a child package that is used in multiple parent packages For

example, if you want to modify a child package that is used in five parent packages, you

can modify your child package only once and all the five parent packages will pick up

the modified child package

While developing your packages with the modular design concept, you can reuse

packages developed as modules to perform specific functions in other packages You

may, for example, develop a send mail package that reads from a table and sends a

broadcast mail to all the members in the table This functionality can be used in any

package that requires sending mails Another example could be auditing for which

you can develop a package that records environment values and variable values to an

audit table You can then attach this package as a child package in all of your packages

wherever you want to record auditing information

Using the Execute Package task, you can have better security controls in place You

can control access to sensitive data by dividing your package into packages that can have public access and other packages that can be accessed by administrators or management

only For example, you can separate out salary processing package from your main SSIS