SQL Server Tacklebox- P14 ppt

Minimally logged transactions When bulk loading data using BCP or SSIS, it is important to know how this massive import of data will effect data and log growth.. If the database to whic

data from source to target can accommodate, and on the speed of your network link

The two bulk transfer tools that we'll consider here are:

• Bulk Copy Program (BCP) – This tool has been around for nearly as

long as SQL Server itself DBAs have a hard time giving it up It is a command line tool and, if speed of data loading is your main criteria, it is still hard to beat There are several caveats to its use, though, which I will cover

• SQL Server Integration Services (SSIS) – I have found that SSIS is

one of the best choices for moving data, especially in terms of cost, and

in situations where near real-time data integration is not a requirement, such as you may achieve with native replication or Change Data Capture technologies Transforming data is also a chore that SSIS handles very well, which is perfect for data warehousing I will show how to use SSIS

to load data from a source to destination, and watch the data as it flows through the process

Whether you choose to use BCP or SSIS will depend on the exact nature of the request Typically, I will choose BCP if I receive a one-time request to move or copy a single large table, with millions of records BCP can output data based on a custom query, so it is also good for dumping data to fulfill one-off requests for reports, or for downstream analysis

SSIS adds a level of complexity to such ad-hoc requests, because DBAs are then forced to "design" a solution graphically In addition, many old school DBAs simply prefer the command line comfort of BCP I am not sure how many old school DBAs remain, but as long as Microsoft continues to distribute BCP.exe, I will continue to use it and write about it, for its simple and fast interface

SSIS has come a long way from its forebear, Data Transformation Services (DTS) and, in comparison to BCP, can be a bit daunting for the uninitiated DBA However, I turn to it often when requested to provide data migration solutions, especially when I know there may be data transformations or aggregations to perform, before loading the data into a data warehouse environment SSIS packages are easy to deploy and schedule, and Microsoft continues to add functionality to the SSIS design environment making it easy for developers to control the flow of processing data at many points Like BCP, SSIS packages provide a way to import and export data from flat files, but with SSIS you are not limited to flat files Essentially any ODBC or OLEDB connection becomes a data source Bulk data loads are also supported; they are referred to as "fast Load" in SSIS vernacular

Over the coming section, I'll present some sample solutions using each of these tools First, however, we need to discuss briefly the concept of minimally logged transactions

Minimally logged transactions

When bulk loading data using BCP or SSIS, it is important to know how this massive import of data will effect data and log growth In this regard, it is important to review the concept of "minimally logged" transactions If the database to which you are bulk loading the data is using the Full recovery model, then such operations will be "fully logged" In other words, the transaction log will maintain a record for each and every inserted record or batch This transaction logging, in conjunction with your database backups, allows for point-in-time recovery of the database

However, if you were to load in 50 million records into a database in Full recovery mode this could eventually be a nightmare for the DBA Transactions in the log file for a Full recovery database are only ever removed from the log upon a transaction log backup and so, in the absence of frequent log backups, log file growth would spiral out of control

As such, you may consider switching to one of the other available recovery models, Simple or Bulk-logged, for the duration of the bulk import operation In these recovery modes, such operations (and a few others) are only minimally logged Enough information is stored to recover the transaction, but the information needed to support point-in-time recovery is not written to the transaction log Note, however, that there are a few caveats to this exemption from full logging If, for example, there is a clustered index on the table that you are bulk loading, all transactions will be fully logged

So, for example, in order to minimize logging for bulk activities, such as those used by BCP.exe, you can temporarily switch from Full recovery mode to Bulk-logged mode, while retaining the ability to back up the transaction log One downside of Bulk-logged mode, however, is that you lose the ability to restore to a point in time if there are any bulk transactions, though you can still restore the entire transaction log in Bulk-logged mode

Alternatively, you can set the database to Simple mode, in which bulk operations are also minimally logged By definition, the Simple mode does not support

point-in time recovery, spoint-ince the transaction log cannot be backed up, and is truncated each time a checkpoint is issued for the database However, this "truncate on checkpoint" process does have the benefit that the log is continually freed of committed transactions, and will not grow indefinitely

The dangers of rampant log file growth can be mitigated to some extent by committing bulk update, inserts or delete transactions in batches, say every 100,000 records In BCP, for example, you can control the batch size using the batch size flag This is a good practice regardless of recovery model, as it means that the committed transaction can be removed from the log file, either via a log backup or a checkpoint truncate

The model in normal use for a given database will depend largely on your organization's SLAs (Service Level Agreements) on data availability If point-in-time recovery is not a requirements, than I would recommend using the Simple recovery model, in most cases Your bulk operations will be minimally logged, and you can perform Full and Differential backups as required to meet the SLA However, if recovering to a point in time is important, then your databases will need to be in Full recovery mode In this case, I'd recommend switching to Bulk logged mode for bulk operations, performing a full backup after bulk loading the data and then subsequently switching back to Full recovery and continuing log backups from that point

NOTE

I cover many tips and tricks for monitoring file growth in Chapter 4, on managing space

BCP.EXE

BCP has been a favorite of command line-driven DBAs ever since it was introduced in SQL Server 6.5 It has retained its popularity in spite of the introduction of smarter, prettier new tools with flashy graphical interfaces and the seeming ability to make data move just by giving it a frightening glare I have used BCP for many tasks, either ad hoc, one-off requests or daily scheduled loads Of course, other tools and technologies such as SSIS and log shipping shine in their own right and make our lives easier, but there is something romantic about BCP.exe and it cannot be overlooked when choosing a data movement solution for your organization

Basic BCP

Let's see how to use BCP to migrate data from our SQL_Conn table in the

DBA_Rep database We'll dump the 58K rows that currently exist in my copy of the table to a text file, and then use a script to repeatedly load data from the file back into the same SQL_Conn table, until we have 1 million rows

Knowing that the table SQL_Conn is a heap, meaning that there are currently no indexes defined for the table, I rest easy knowing that I should be minimally

logging transactions, as long as the database is set for the Bulk logged or Simple recovery model

With BCP, just like with SSIS dataflow, data is either going in or coming out Listing 3.2 shows the BCP output statement, to copy all of the data rows from the

SQL_Conn table on a local SQL Server, the default if not specified, into a text file

bcp dba_rep SQL_Conn out "C:\Writing\Simple Talk

Book\Ch3\Out1.txt" -T –n

Listing 3.2: BCP output statement

After the bcp command, we define the source table, in this case

dba_rep SQL_Conn Next, we specify out, telling BCP to output the contents of the table to a file, in this case, "C:\Writing\Simple Talk Book\Ch3\Out1.txt" Finally, the -T tells BCP to use a trusted connection and -n instructs BCP to use native output as opposed to character format, the latter being the default

Native output is recommended for transferring data from one SQL Server instance to another, as it uses the native data types of a database If you are using identical tables, when transferring data from one server to another or from one table to another, then the native option avoids unnecessary conversion from one character format to another

Figure 3.1 shows a BCP command line execution of this statement, dumping all

58040 records out of the the SQL_Conn table

According to Figure 3.1, BCP dumped 17 thousand records per second in a total

of 3344 milliseconds, or roughly 3 seconds I would say, from first glance, that this

is fast The only way to know is to add more data to this table and see how the times change Remember that at this point, we are just performing a straight

"dump" of the table and the speed of this operation won't be affected by the lack

of indexes on the source table However, will this lack of indexes affect the speed when a defined query is used to determine the output? As with any process, it is fairly easy to test, as you will see

Let's keep in mind that we are timing how fast we can dump data out of this sample table, which in the real world may contain banking, healthcare or other types of business critical data 58 thousand is actually a miniscule number of records in the real world, where millions of records is the norm So let's simulate a million records so that we may understand how this solution scales in terms of time and space I roughly equate 1 million records to 1 Gigabyte of space on disk,

so as you are dumping large amounts of data, it is important to consider how much space is required for the flat file and if the file will be created locally or on a network share The latter, of course, will increase the amount of time for both dumping and loading data

Figure 3.1: Dumping 58K records out of the SQL_Conn table

In order to simulate a million or more records, we can load up the 58,000 records into a table multiple times so that we cross the plateau of 1 million records I have created a batch file to do this, which is shown in Listing 3.3 In this case, I am loading the data back into the same table from which it came, SQL_Conn

set n=%1

set i= 1

:loop

bcp dba_rep SQL_Conn in

C:\Writing\Simple Talk Book\Ch3\Out1.txt"

-n -b 50000 –T -h "TABLOCK"

if %i% == %n% goto end

set /a i=i+1

goto loop

:end

Listing 3.3: Batch file to load 1 million records from 58,000

You will see that the main difference between this BCP statement and the previous one is that instead of out I am specifying in as the clause, meaning that