Hands-On Microsoft SQL Server 2008 Integration Services part 18 pot

In the next exercise, you will unzip the downloaded files using the Execute Process task, which will run a batch file to do the trick... In the last chapter, you used a Foreach Loop Cont

In this Hands-On exercise, you learned to use the FTP task to download multiple files from a remote FTP server You have hard-coded file paths for the RemotePath and LocalPath fields in this exercise However, you can use variables to change these folders dynamically during run time and download or upload files to different folders each time the package is run For example, you can download files to the folders derived on the basis of date—for instance, the folder name includes the date as a part of the name (such as Zip20091011) So, in that case, when the package is run, the path variable is calculated dynamically based on the current date and loads the files to the current date folder Variables are not the only means to derive values dynamically SSIS enables you

to derive values dynamically at run time, based on the outcome or values generated by other tasks, using property expressions as well that do not necessarily contain variables

in their expressions You have been introduced to the property expressions in Chapter 3 and will learn more about them in Chapter 8

Execute Process Task

Generally enterprises have home-grown or custom-developed applications to apply business processes to the information they deal with These applications might be to make sure that the information goes to right people in the right format after corporate standards are applied on to it, or it might be a custom application to transform a mainframe encrypted and compressed (such as EBCDIC format) file into a standard text format file The company might wish to reuse the functionality that already exists and has been accepted within the corporate setting Integration Services allows you

to leverage your existing investment and bring those processes in the workflow within the packages by using the Execute Process task With this task, you can run a business application or a batch file to perform specific business functions that already exist or can’t be developed easily in SSIS; the good thing about this is that the output from running the business application or batch file can then be included in the SSIS package workflow You could unzip files, call custom programs to break a huge file into smaller files that can be easily handled for transformation operations, run a custom report generation application from within SSIS, and then use a Send Mail task to distribute those reports To keep it simple, we will use this task to unzip the files you have downloaded from the FTP server in the preceding Hands-On exercise

You have begun your journey to understand the SSIS workflow In the last Hands-On exercise, you downloaded zipped files from an FTP server in the C:\SSIS\ downloads folder In the next exercise, you will unzip the downloaded files using the Execute Process task, which will run a batch file to do the trick

Hands-On: Expanding Downloaded Files

Adventure Works Bikes dealers’ reports need to be downloaded and extracted so that

they can be imported into the SQL Server database You have already downloaded

these files; here you will extract flat files from the downloaded files

Method

You have downloaded zipped files DealerSales01.zip and DealerSales02.zip in the

C:\SSIS\downloads folder but can’t use these files, as SSIS can’t read zipped files

You need to extract flat files from the zipped files to be able to import them into SQL

Server using SSIS You will run a batch file to extract the flat files from the zipped

files and will deploy an Execute Process task to run this batch file at run time Because

you want to unzip more than one file, you will enumerate the files you want to unzip;

an enumeration function will be provided by a Foreach Loop Container In the last

chapter, you used a Foreach Loop Container with the Foreach ADO enumerator to

enumerate over rows; now you will be using the Foreach File enumerator to enumerate

over files The steps for this will be:

Create a batch file to expand the downloaded Zip file

c

Use a Foreach Loop Container to enumerate over multiple Zip files

c

Add an Execute Process task to call a batch file from within the SSIS package to

c

extract the flat files

Exercise (Creating Batch File)

The downloaded files have been compressed using a Freezip freeware package that

has been included in the distribution software provided for this book Go to C:\SSIS\

Freeware\Freezip to find the software files there However, if you use some other

software utility to compress files, you can use a command-line interface of that utility to unzip files in the same manner as shown in this exercise

Open a blank text file using Notepad and type the following commands:

DEL C:\SSIS\downloads\%1.txt

C:\SSIS\Freeware\Freezip\UNZIP %1.zip %1.txt

The first line deletes the previously present text file, and the second line extracts a

text file from the Zip file specified as an argument in the command Save the file as

C:\SSIS\RawFiles\ExtractFiles.bat Make sure that the file has been saved with the

.bat extension and not with the txt extension This file should already be available in

the RawFiles folder, as it has been included in the distribution software provided for

this book

Exercise (Enumerating Multiple Zipped Files)

You will add and configure a Foreach Loop Container to enumerate over the Zip files

in the downloads folder and populate the fname variable with the filename.

1 Start BIDS and open the Control Flow Tasks project Go to Solution Explorer and right-click the SSIS Packages folder in the Control Flow Tasks project and choose New SSIS Package from the context menu This will add a new package called Package1.dtsx in the project

2 Right-click Package1.dtsx and rename it as Expanding downloaded files.dtsx.

3 Drop a Foreach Loop Container from the Toolbox onto the designer surface and open the editor by double-clicking it Type the following in the General page:

Name Enumerating Zip Files

Description This task enumerates Zip files in the C:\SSIS\downloads folder

4 Go to Collection page and make sure Foreach File Enumerator is selected in the Enumerator field

5 In the Enumerator configuration area, specify C:\SSIS\downloads in the Folder field either by typing it in directly or by using the Browse button to select the folder You can enumerate multiple files of the same type with the Foreach Loop Container at one time For this, you can use the Files field to limit or select precisely the files of one type you want to access in a folder containing different types of files In the Files

field, type *.zip to specify the types of files you want the Foreach Loop Container

to enumerate on (refer Figure 5-3) Use the Files field to limit or select precisely the files you want to access in a folder containing different types of files

6 The most interesting bit in this page is the Retrieve File Name section This is where you will choose how the filename should be retrieved from three options Before you jump into selecting one of the options, you must understand what these options are and how they work The filename is treated in SSIS as consisting of three parts: the path pointing to the location where the file might be stored, the name portion of the file, and the extension of the file indicating its type

Fully Qualified This option will return the path, the filename, and the extension all as a single

string The path portion of the filename can be a full folder path in a universal naming convention (UNC) or absolute form For example, the fully qualified name

retrieved can be \\ComputerName\Sales\DealerSales01.zip, or it can be in

the form C:\SSIS\downloads\DealerSales01.zip, depending on what you have specified in the Folder field

Name And Extension Choosing this option returns the name portion of the filename along with its

extension—for example, DealerSales01.zip

Name Only Choosing this option will return only the name portion of the filename—for

example, DealerSales01

Select Name only, as you will need only the name of the file to be passed as an

argument to the batch file

7 Go to the Variable Mappings page, click in the Variable field, and select <New

Variable…> to create a new variable Name the variable fname in the Expanding

Download Files Container with the string type value and return to the Foreach

Loop Editor Note that the Index field has automatically been allocated a value

of 0 for this variable Make sure that you type the variable name in lowercase—

fname—as the variables are case sensitive Click OK to complete the Foreach

Loop Container’s configuration

Figure 5-3 Configuring the Foreach Loop Container with the Foreach file enumerator

Exercise (Calling Batch File Using Execute Process Task)

Now you will add Execute Process task inside the Foreach Loop Container This

task will use the fname variable populated by Foreach Loop Container to pass as an

argument to the ExtractFiles.bat file

8 Drag and drop the Execute Process task from the Toolbox within the Foreach Loop Container and double-click it to open the Execute Process Task Editor Type the following in the General page of the task editor:

Name Call ExtractFiles batch file

Description This task extracts DealerSales flat files

9 Go to Process page to configure options for this task Verify that the RequireFullFileName field has the default True selected This means that the task will fail if it cannot find the batch file at specified location

10. In the Executable field, specify C:\SSIS\RawFiles\ExtractFiles.bat to point the task to the batch file

11. You need to specify the filename as an argument to the batch file The filename

argument has been populated in the fname variable in the earlier part of this

exercise However, Arguments field doesn’t allow use of a variable, so you will

be using property expressions in the next step to attach the fname variable to the Arguments field Leave the Arguments field blank for the moment and in the

WorkingDirectory field, specify C:\SSIS\downloads, as shown in Figure 5-4.

In the next three fields, you can use variables for the input, output, or error values

as explained here:

c StandardInputVariable Specify a variable to provide the input to the

process

�c StandardOutputVariable Specify a variable to capture the output of the

process

c StandardErrorVariable Specify a variable to capture the error output of the

process

You are not using these fields for this exercise, so leave them blank Using the FailTaskIfReturnCodeIsNotSuccessValue field, you can configure the task to fail

if the process exit code does not match the value you provide in the SuccessValue field Also, you can specify a time-out value in seconds and choose to terminate the task after time-out period Finally, if it is important, you can choose the window style in which the task starts a process Leave these fields set at their

default values and move on to the Expressions page.

12. In this step, you will pass the value of the fname variable to the Arguments

field using property expressions Property expressions are explained in detail in

Chapter 8, but here for a quick introduction, property expressions will apply the

run-time values of the fname variable on to the Arguments property Click the

ellipsis button in the Expressions field to open the Property Expressions Editor

Click in the Property field and select Arguments from the drop-down list Click

the ellipsis button in the Expression field to open an Expression Builder dialog

box Expand Variables in the left pane of the window and locate the User::fname

variable in the list Drag this variable and drop it in the Expression box Click OK three times to close the task editor

13. Press f5 to start debugging the task You should see the tasks turning quickly

from yellow to green (see Figure 5-5)

Figure 5-4 Configuration of the Execute Process task

13. Press ctrl-shift-s to save all the files in this project and then choose File | Close Project

14. Go to the downloads folder and see that the DealerSales01.txt and DealerSales02 .txt files have been extracted

Review

You have learned how to use the Execute Process task to expand the zipped files in this Hands-On exercise Now you know that SSIS provides an option to run the custom-built applications that you need to use for your day-to-day functions Also, not only can you run custom-built applications, but you can also use the results of running such an application

by using the StandardOutputVariable property and use this value in other places in the package such as building a workflow expression

Figure 5-5 Executing the Expanding downloaded files.dtsx package

File System Task

The File System task allows you to perform operations on the files and directories in

the file system You will be using this task in your packages to perform operations such

as copy, delete, and move on files and folders If you have used DTS 2000, you might

have written ActiveX code to rename, copy, or move files or folders Though you may

still have to write a piece of code in the Script task to perform some of the complex

operations, most of your requirements for performing operations on files and folders

can be met by using the File System task and hence less code to maintain

In the Task Editor of File System task, you will see an Operation field that offers a

drop-down list of operations it can perform Following are the brief descriptions of the

operations that the File System task can perform:

Copy directory

c Copies a folder from one location to another

Copy file

c Copies a file from one location to another

Create directory

c Creates a directory in a specified location

Delete directory

c Deletes a directory from a specified location

Delete directory content

c Deletes all the files and folders in the specified directory

Delete file

c Deletes the specified file

Move directory

c Moves a folder from one location to another

Move file

c Moves a file from one location to another location

Rename file

c Renames a file in the specified location You can actually move

the file to a new location while renaming by specifying a new path in the name,

though you cannot rename the file while moving it to a new location

Set attributes

c Sets Archive, Hidden, ReadOnly, and System attributes on the

files and folders If you do not specify any attribute in your selections, the Normal

attribute will be applied, which is exclusive to all the other attributes

The File System task has a dynamic layout in which the available fields change

depending on the operation you choose For example, the Delete Directory, Delete

Directory Content, and Delete File Operations do not use a destination, and hence

the DestinationConnection field doesn’t appear when you choose any of these

operations The SourceConnection, however, is used by all of the operations The

SourceConnection and the DestinationConnection can be specified by using a File

Connection Manager to refer to a file or a folder or by providing a variable that

contains the connection strings

The following Hands-On exercise for this task will help you understand the configuration options In the earlier exercises, you have downloaded and unzipped the dealer sales files and hence have compressed Zip files and uncompressed text files

in the C:\SSIS\downloads folder Your next problem is to archive the Zip files for historical purposes

Hands-On: Archiving Downloaded Files

The downloaded reports of dealers of Adventure Works Bikes are to be archived for historical purposes You want to keep the downloaded zipped files in the Archive folder

Method

In this exercise, you will use the Foreach Loop Container and the File System task to copy these files one by one to the C:\SSIS\downloads\Archive folder The step-by-step method is as follows:

Configure the Foreach Loop Container to enumerate over files and house the File c

System task in it Your File System task will copy one file with each iteration to the Archive folder

Configure the File System task to move files to the Archive folder one by one with c

each iteration of the Foreach Loop Container

Exercise (Configure Foreach Loop Container)

In first part of the exercise you will configure the Foreach Loop Container to enumerate files using Foreach File enumerator and populate a variable with the filename This variable will be used by File System task to determine the name of the file to be copied over

1 Open the Control Flow Tasks project in BIDS Go to the Solution Explorer, right-click the SSIS Packages folder, and choose New SSIS Package from the context menu This will add a new package, Package1.dtsx, to the project

2 Right-click Package1.dtsx and rename it as Archiving downloaded files.dtsx.

3 From the Toolbox, drag and drop the Foreach Loop Container onto the Control Flow panel

4 Make sure that the Foreach Loop Container is selected; open the Variables window from the default position on the left side of the BIDS interface Alternatively, click the Foreach Loop Container and choose View | Other Windows | Variables

Click Add Variable, the left-most button in the toolbar of the Variables window

Specify the following for this variable:

Scope Foreach Loop Container

The value you’ve assigned to fname variable will be changed at run time when the

Foreach Loop Container reads the filename and populates this variable during

each iteration The value assigned is just a placeholder value, so you can realize

the Fname variable changes values dynamically Last, be careful with the case of

the Name field, as SSIS variables are case-sensitive

5 Right-click the Foreach Loop Container and choose Edit from the context menu

to open the Foreach Loop Editor In the General page of the container, type the

following:

Name Enumerating Files

Description This container enumerates the files and populates the fname variable

6 Go to the Collection page Make sure it shows the Foreach File Enumerator in

the Enumerator field and specify the options shown in Figure 5-6

Enumerator Foreach File Enumerator

Folder C:\SSIS\downloads

Retrieve file name Fully qualified

7 Move on to Variable Mappings page You will map fname variable to the filename

retrieved by the Foreach Loop Container so that later on in the package you can

get the filename by the fname variable Click in the highlighted area under the

Variable column and then click the down arrow that appears From the

drop-down list, select the variable User::fname, configured earlier in the exercise A 0 will

appear under the Index field Index indicates the position of the string within the

collection As you will be accessing only one filename at a time, Index will be set

to 0 Click OK to close the Foreach Loop Editor