SQL server MVP Deep dives Volume 1

In this book, the world’s leading practitioners of SQL Server present a collection of articles on techniques and best practices for SQL Server development and administration based on many years of combined experience. The 53 MVPs who contributed to the book each picked an area of special interest to them and shared their insights and practical knowhow with you. The topics covered will appeal to a broad range of readers with varied levels of SQL Server experience, from beginner to advanced.

MVP CONTRIBUTORS

John Baird ● Bob Beauchemin ● Itzik Ben-Gan ● Glenn Berry ● Aaron Bertrand ● Phil Brammer ● Robert C Cain ● Michael Coles ● John Paul Cook ● Hilary Cotter ●Louis Davidson ● Christopher Fairbairn ● Rob Farley ● Denis Gobo ● Bill Graziano ● Dan Guzman ● Paul Ibison ● Tibor Karaszi ● Kathi Kellenberger ● Don Kiely ●Kevin Kline ● Hugo Kornelis ● Alex Kuznetsov ● Matija Lah ● Cristian Lefter ● Andy Leonard ● Greg Linwood ● Bruce Loehle-Conger ● Brad McGehee ● Paul Nielsen ●Pawel Potasinski ● Matthew Roche ● Dejan Sarka ● Edwin Sarmiento ● Gail Shaw ● Linchi Shea ● Richard Siddaway ● Jasper Smith ● Erland Sommarskog ●Scott Stauffer ● Tom van Stiphout ● Gert-Jan Strik ● Ron Talmage ● William R Vaughn ● Joe Webb ● John Welch ● Erin Welker ● Allen White

DEEP DIVES

E DITED BY

Paul Nielsen Kalen Delaney Greg Low Adam Machanic Paul S Randal Kimberly L Tripp

SQL Server MVP

Deep Dives

www.manning.com The publisher offers discounts on this book when ordered in quantity For more information, please contact

Special Sales Department

Manning Publications Co

Sound View Court 3B

Greenwich, CT 06830

Email: orders@manning.com

©2010 by Manning Publications Co All rights reserved

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in the book, and Manning

Publications was aware of a trademark claim, the designations have been printed in initial caps

or all caps

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine

Development editor: Jeff BleielManning Publications Co Lead copyeditor: Andy Carroll

ISBN 978-1-935182-04-7

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 – MAL – 14 13 12 11 10 09

Pawel Potasinski 19 Matthew Roche 57 Dejan Sarka 54 Edwin Sarmiento 35 Gail Shaw 42 Linchi Shea 48 Richard Siddaway 26 Jasper Smith 28 Erland Sommarskog 17 Scott Stauffer 38 Tom van Stiphout 14 Gert-Jan Strik 46 Ron Talmage 31 William R Vaughn 18, 51 Joe Webb 30, 44

John Welch 56 Erin Welker 55 Allen White 27

Technical Editor

Rod Colledge

To all children traumatized by the horrors of war and War Child’s efforts to help children be children again

brief contents

1 ƒ Louis and Paul’s 10 key relational database design ideas 3

4 ƒ Set-based iteration, the third alternative 43

14 ƒ Simil: an algorithm to look for similar strings 200

18 ƒ Getting and staying connected—or not 255

31 ƒ Some practical issues in table partitioning 413

55 ƒ To aggregate or not to aggregate—is there really a question? 700

contents

preface xxxv acknowledgments xxxviii about War Child xxxix about this book xli about SQL Server MVPs xliii

P ART 1 D ATABASE DESIGN AND ARCHITECTURE 1

EDITED BY PAUL NIELSEN

PAUL NIELSEN AND LOUIS DAVIDSON

2 Keys are key 4

4 Class <> table 5

5 Data drives design 6

6 Sets good, cursors bad 6

7 Properly type data 7

8 Extensibility through encapsulation 7

9 Spaghetti is food, not code 7

2 SQL Server tools for maintaining data integrity 11

LOUIS DAVIDSON

Data types 13 ■ NULL specification 13 ■ Uniqueness constraints 14 ■ Filtered unique indexes 15 ■ Foreign key constraints 16 ■ Check constraints 17 ■ Triggers 20

HUGO KORNELIS

First step: finding single-attribute dependencies 31 ■ Second step: finding two-attribute dependencies 36 ■ Further steps: three-and-more-attribute dependencies 39 ■ What if I have some independent attributes left? 39

P ART 2 D ATABASE D EVELOPMENT 41

EDITED BY ADAM MACHANIC

HUGO KORNELIS

Declarative (set-based) code 44 ■ Iterative (cursor-based) code 45

Gaps—solution 1 using subqueries 62 ■ Gaps—solution 2 using subqueries 63 ■ Gaps—solution 3 using ranking functions 64 Gaps—solution 4 using cursors 64 ■ Performance summary for gaps solutions 65

Islands—solution 1 using subqueries and ranking calculations 66

Islands—solution 2 using group identifier based on subqueries 67

Islands—solution 3 using group identifier based on ranking

calculations 68 ■ Islands—solution 4 using cursors 69

Variation on the islands problem 69 ■ Performance summary

for islands solutions 71

BILL GRAZIANO

Returning information about the error 74 ■ Generate your own

errors using RAISERROR 76 ■ Nesting TRY CATCH

blocks 77 ■ TRY CATCH and transactions 79

Handling SQL Server messages on the client 82

A sample query 89 ■ The appearance of most queries 90

When the pattern doesn’t apply 90 ■ How to read a FROM

clause 91 ■ When the pattern can’t apply 91 ■ Writing the FROM clause clearly the first time 92

Filtering with the ON clause 92

The different filters of the SELECT statement 93 ■ Filtering out

the matches 93

Simplification using views 96

8 What makes a bulk insert a minimally logged operation? 102

DENIS GOBO

Creating the file to import 104 Creating the tables to store the data 105

KATHI KELLENBERGER

Why does SQL Server remove the DTD from my XML data? 138 How do I preserve whitespace in my XML? 139 ■ Why am I getting strange characters in my XML? 139 ■ How do I query XML data? 141 ■ How do I query a single value from my XML data? 141 ■ How do I shred XML data? 142

How do I specify an XML namespace in my XQuery queries? 145 How do I get all element names and values from my XML

document? 146 ■ How do I load XML documents from the filesystem? 147

MATIJA LAH

The concept 150 ■ The logical model 152 ■ The physical model 154 ■ The database 155 ■ The XML Schema 155

Preparing the inbound data flow 160 ■ Importing the data 163 Exporting the data 171

ROBERT C CAIN

Foundations of full-text searching 176

Creating and maintaining full-text indexes 178

Creating the full-text index 178 ■ Maintaining full-text indexes 180

Querying full-text indexes 181

Basic searches 181 ■ FORMSOF 182 ■ Phrases, NEAR, OR, and prefixed terms 183 ■ Ranking 184

Custom thesaurus 186 ■ Stopwords and stoplists 189

Basic queries to discover what catalogs, indexes, and columns exist 192 ■ Advanced queries 194 ■ The keywords 196

14 Simil: an algorithm to look for similar strings 200

TOM VAN STIPHOUT

DON KIELY

Using a DataTable 227 ■ Using a DbDataReader 228 Using TVPs to enter orders 229

ERLAND SOMMARSKOG

Using the LIKE operator—an important observation 236 Using a binary collation 237

The fragments_persons table 237 ■ Writing the search procedure 238 ■ Keeping the index and the statistics updated 241 ■ What is the overhead? 242

Building the lists 244 ■ Unwrapping the lists 244 The fragments_personlists table 245 ■ Loading the table 245

A search procedure 248 ■ Keeping the lists updated 249

The initial setup 250 ■ Searching with the bitmask 251 Adapting the bitmask to the data 251 ■ Performance and overhead 252 ■ The big bitmask 253

WILLIAM VAUGHN

Testing for network availability 257 ■ Managing the SQL Server instance state 258 ■ Finding visible SQL Server instances 259

Trusted or untrusted security? 269

Using trusted or integrated security 270 ■ ASP.NET considerations 270 ■ Using SQL Server authentication 271

19 Extending your productivity in SSMS and Query Analyzer 277

PAWEL POTASINSKI

Creating your custom utility to use with

Generating a calendar on the fly 284 ■ Splitting strings with a numbers table 286

JOHN BAIRD

Client synchronization provider 306 ■ Server synchronization provider 306 ■ Synchronization adapter 307

Synchronization agent 308 ■ Using MSF 310

Comparison 316

P ART 3 D ATABASE A DMINISTRATION 319

EDITED BY PAUL S RANDAL

AND KIMBERLY L TRIPP

Versions and service packs 331 ■ One or several schedules per plan? 331 ■ Wizard dialogs 332 ■ Task types 332 Check database integrity task 333 ■ Shrink database task 333 Reorganize index task 334 ■ Rebuild index task 335

Update statistics task 335 ■ History cleanup task 336 Execute SQL Server Agent job task 338 ■ Back up database task 338 ■ Maintenance cleanup task 340 ■ Select reporting options 341 ■ Execute T-SQL statement task 341

RICHARD SIDDAWAY

PowerShell features 344 ■ PowerShell issues and solutions 345

Data access 346 ■ Administration 347

PowerShell with SQL Server 2008 348

Overview of audit infrastructure 365

Server audit objects 365 ■ Server audit specification objects 366 Database audit specification objects 366

Server audit specifications 372

Creating server audit specifications 372 ■ Viewing audit events 374

Database audit specifications 376

Creating database audit specifications 376

AARON BERTRAND

sys.dm_os_sys_info 383 ■ sys.dm_exec_sessions, sys.dm_exec_requests, and sys.dm_exec_connections 383 sys.dm_exec_sql_text 384 ■ sys.dm_exec_query_stats 384

sys.dm_exec_procedure_stats 384 sys.dm_db_index_usage_stats 385 sys.dm_db_missing_index_details, sys.dm_db_missing_index_groups, and sys.dm_db_missing_index_group_stats 385

sys.dm_os_performance_counters 385 sys.dm_db_partition_stats 386 sys.dm_db_index_physical_stats 386 sys.dm_sql_referenced_entities 387

Setting up a utility database 387

Some interesting applications of my favorite DMVs 390

A more refined sp_who or sp_who2 390 ■ Getting statistics for stored procedures (SQL Server 2008 only) 393 ■ Finding unused stored procedures (SQL Server 2008 only) 394 ■ Finding inefficient and unused indexes 396 ■ Finding inefficient queries 398 ■ Finding missing indexes 399

Using DBCC CLEANTABLE to reclaim unused table

RON TALMAGE

Drilling down: using SPLIT and MERGE 417 ■ Drilling down: using SWITCH 418

32 Partitioning for manageability (and maybe performance) 421

DAN GUZMAN

Boundary time values 429 ■ SPLIT and MERGE performance 430 ■ Update statistics after SWITCH 430 Shared partition functions and schemes 430

Restoring for extraction only 437 ■ Restoring for production use 437 ■ Restoring for production use—step by step 439

Case study of moving data with database mirroring 459 Lessons learned from case study 460

EDWIN SARMIENTO

Creating a batch file 466

Remove redundant pre-snapshot and post-snapshot scripts 481 Replace merge -EXCHANGETYPE parameters 482

Large numbers of push subscriptions 492

Optimal settings for replication 492

CommitBatchSize and CommitBatchThreshold 493 ■ Update proc 494 ■ SubscriptionStreams 495

38 Successfully implementing Kerberos delegation 496

Service principle names 498 ■ Constrained delegation 501

Configuring the Active Directory 503 ■ Configuring the client tier 505 ■ Configuring the web tier 508 ■ Configuring the data tier 510

Downloading and installing DelegConfig 512 ■ Running DelegConfig from the web tier 512 ■ Running DelegConfig from the client tier 512

Resources to assist in more complex infrastructures 512

JOHN PAUL COOK

Virtualization architecture 519 Benefits of isolation 520

Configuring disks 521 ■ CPU configuration 523 Configuring networking 524 ■ Memory configuration 525

Addressing clock drift issues 525

Advantages of physical to virtual migration 526

P ART 4 P ERFORMANCE T UNING AND O PTIMIZATION 529

EDITED BY KALEN DELANEY

ROB FARLEY

ALEX KUZNETSOV

Some rules of thumb about indexes aren’t true for covering

One index can both cover queries and implement

Preallocating tempdb space and avoiding use of autogrowth 561 Don’t shrink tempdb if you don’t need to 562 ■ Dividing tempdb among multiple physical files 562 ■ Moving tempdb to a disk separate from your other databases 563 ■ Locating tempdb on a fast I/O

subsystem 563 ■ Adding RAM to your SQL server

instance 564 ■ Using SQL Server 2008 transparent data encryption 564 ■ Leaving auto create statistics and auto update statistics on 564 ■ Verifying CHECKSUM for SQL Server 2008 564

JOE WEBB

Finding a specific row 567

KEVIN KLINE

Best practices using PerfMon 576 ■ Best practices using Profiler 578 ■ A correlated view of performance 580

GERT-JAN STRIK

Low correlation 584

GLENN BERRY

Looking at top waits at the instance level 591

LINCHI SHEA

Random or sequential I/Os and disk performance

Scenario 1: constant checkpoints 613 ■ Scenario 2: NOLOCK and faster query processing 614 ■ Scenario 3: read-ahead reads 615 Scenario 4: index fragmentation 616

Managing report parameters 650 ■ Deploying your report 651

What’s in Reporting Services 2008 for developers? 654

Virtual directory changes 655 ■ Using SQL Server Reporting Services Configuration Manager 655 ■ Exporting reports 656

Enabling My Reports 656

Stored procedures and temp tables 663 ■ Excel merged cell solution 663 ■ Excel web queries and reports 664 ■ HTML or Word documents 667 ■ Server portability 667 ■ Embedding T-SQL

What are these solutions used for? 670

SQL Server Audit 673 ■ Change tracking 676 Change data capture 681

DEJAN SARKA

Data mining projects 689 ■ Data overview and preparation 690 SSAS 2008 data mining algorithms 690

Harvesting the results 693

Viewing the models 694 ■ Evaluating the models 695 Creating prediction queries 697

55 To aggregate or not to aggregate—is there really a question? 700

Introduction to the Data Profiling task 711

Types of profiles 711 ■ Input to the task 712 ■ Output from the task 713 ■ Constraints of the Data Profiling task 715

Changing the database 716 ■ Altering the profile requests 716 Setting the ProfileInputXml property 717

Excluding data based on quality 719 ■ Adjusting rules dynamically 719

Capturing the output 720 ■ Using SSIS XML functionality 721 Using scripts 723 ■ Incorporating the values in the package 723

Expressions in the data flow 734

PHIL BRAMMER

Source acquisition performance 744 ■ Data transformation performance 745 ■ Destination performance 746 ■ Lookup transformation performance 747 ■ General data flow performance 748

Creating the new BIDS project 754 ■ Defining the lookup transformation 755 ■ Setting the lookup transformation behavior 757

index 763

preface

Each year Microsoft invites all the MVPs from every technology and country to mond for an MVP Summit—all top secret—“don’t tweet what you see!” During theMVP Summit, each product team holds a series of presentations where they explaintheir technologies, share their vision, and listen to some honest feedback

At the 2007 MVP Summit in Seattle, Bill Gates presented his vision of the future ofcomputing to the MVPs and then took questions for about an hour I really enjoy thesedialogues I get the sense that if BillG wasn’t the founder of Microsoft, he’d make agreat MVP You can tell he likes us as fellow Geeks, and he’s rather candid in the MVPQ&A time It’s one of my favorite parts of the MVP Summit

During the Q&A, the lines at the microphones are far too long to bother to join, so

I daydream a few questions I’d ask BillG:

ƒ As the world’s wealthiest Geek, what can you tell me about your PC?

ƒ Even with all your accomplishments, do you still find the most happiness andsatisfaction with your family?

ƒ Do you play Age of Empires 2, and if so, want to join me in a game?

ƒ Kirk or Picard?

ƒ Can I buy you lunch?

And then I thought of a good, deep, Charlie Rose-type of question: “Centuries fromnow, would you rather be remembered as the guy who put a computer on every desk,

or as the guy who ended malaria and fought the good fight against poverty?”

As I try to guess what BillG might say, the answer is obvious I’m glad that BillG’sintellect and resources are being directed at improving the human condition, and as

an original Windows fan I’m proud of BillG But the answer to my question isboth—Windows has already done as much to fight poverty as will the Bill and MelindaGates Foundation

Toward the end of the Q&A time, which was mostly taken up with technical tions, I was thrilled to hear one of the MVPs ask for his advice as a philanthropist.BillG said that we should all be involved in our communities and give of ourselves increative ways: at the library, at schools, and with charities “Do philanthropy where youare.” This idea of giving of ourselves is central to the MVP community

I’d been noodling for a while with the idea of an MVP charity book to help dren, and I discussed it with my friend, Kalen Delaney, who liked the idea The nextstep was to float the idea past the other SQL Server MVPs and see if it resonated withany of them The response was overwhelming—within a couple of days, about 50MVPs had signed on to the project The question was, would a publisher bite?

My agent, David Fugate of Launch Books, offered to help us for free As Davidworked out how to assemble a contract with 50 authors, with all the author proceedsgoing to charity, Kalen and I pitched the idea to those we know in publishing WhenMichael Stephens of Manning Publications heard about it, he told me that Manningwanted in—that this was exactly the type of project that Manning wanted to publish.Mike liked both the community aspect of the project and the charity goals of the proj-ect Manning also offered us a higher than usual author royalty, because we’re giving

it all to charity

With a book contract under way, Kalen and I recruited Adam Machanic, Greg Low,Paul Randal, and Kimberly Tripp to help us with the editing The book was dividedinto five sections aligned with the five job roles around SQL Server—database archi-tecture and design, database development, database administration, performancetuning and optimization, and business intelligence There was no rigid organization

to the outline—MVPs were simply asked to submit abstracts for chapters that theywanted to write, and those abstracts were directed to the appropriate section editor.This would be a book driven by the MVPs’ individual passions, not by a comprehensivefeature list The section editors selected the best abstracts, but we committed to theidea that every MVP who wanted to contribute to the book could contribute

To select the charity, the MVPs nominated their choice with the restriction that ithad to be a non-sectarian group that benefited children And we wanted to give to asmaller charity—we didn’t just want our contribution to be added to a billion-dollarfund Hugo Kornelis had heard good things about War Child in the Netherlands, andhis nomination was selected in the first round of voting

I’m still amazed that this project came together The 53 MVPs who contributed tothis book did so for numerous personal reasons—a desire to make a difference in chil-dren’s lives, wanting to share knowledge, wanting to participate in a group effort forchange For some of us, it was an expression of our faith Several MVPs told me theywere glad to be a part of a project that reached beyond their typical circles and helpedchildren

If you are reading this book, then you are “rich.” Considering your place in humanhistory, you’re wealthier than most kings of centuries past—you are well educated,your grocery store shelves are full, you have a family doctor For too many in the

world, that is not the case There are communities without clean water, children ing from war, and AIDS orphans who have no family or place to sleep When one pon-ders the immense need and poverty in the world, it’s easy to become overwhelmedwith hopelessness How can a single drop of ink change the color of an ocean? But wehave no other option than to do what we can My philosophy is that of Confucius: “It isbetter to light one small candle than to curse the darkness.”

Even BillG can’t heal the world, but we can each make a difference By buying thisbook, you’ve supported War Child I echo BillG’s words, that we should all find ways to

do philanthropy where we are I encourage you to find the best way you can to make adifference

So, welcome to SQL Server MVP Deep Dives—a collaborative work by 53 passionateMVPs

PAUL NIELSEN Some (not all!) of the SQL Server MVPs contributing to this book were photographed by Bill Vaughn at the 2009 MVP Summit.

To Marjan Bace, our publisher, thank you for helping us refine the concept, porting the project, and partnering with us

To Michael Stephens, Jeff Bleiel, Andy Carroll, Deepak Vohra, Katie Tennant, tie Marsico, Todd Green, Mary Piergies, and the rest of the team at Manning Publica-tions, thanks for believing in this project and enduring with it to the end

Thanks to Bill Vaughn for taking the group photo at the 2009 MVP Summit

A hearty thanks to Rod Colledge (SQL guru from Down Under, of crunch.com fame) for contributing his time as the technical editor Well done!

A thank you to David Fugate of Launch Books for helping us develop the idea andwork out the contract

Many thanks to Ed Lehman, our SQL Server product team MVP Liaison, for hissupport, the many good times, and all the great swag

To our MVP leads through the years, Shawn Aebi, Paul Wehland, Stephen Dybing,Ben Miller, Alison Brooks, and Suzanna Moran, and others internationally, thank youfor supporting our community efforts

To everyone in the Microsoft SQL Server product team, our enthusiastic thanks fordeveloping a product worthy of our passion

A heartfelt thank you to War Child for all you do for the children

And finally, thank you, readers, for supporting our project and helping us supportWar Child

PAUL NIELSEN

about War Child

War Child International is an award-winning charity that provides opportunities andlong-term solutions for war-affected children, focusing on education, strengtheningchildren’s rights, reducing poverty, and fostering self-reliance War Child works inpartnership with local people and organizations to build sustainable programmingthat empowers children and their communities War Child Canada currently providessupport to communities in Afghanistan, Darfur, Sudan, Uganda, Sierra Leone, Demo-cratic Republic of Congo, Ethiopia, Sri Lanka, and Georgia Working closely with theentertainment and tech industries, War Child provides awareness, support, and actionfor children’s rights everywhere

Learn more at www.warchild.org and www.warchild.ca

A letter from the director of War Child follows on the next page