Windows internals

Đây là bộ sách tiếng anh cho dân công nghệ thông tin chuyên về bảo mật,lập trình.Thích hợp cho những ai đam mê về công nghệ thông tin,tìm hiểu về bảo mật và lập trình.

NOTE

Part 2 available Fall 2012

See Table of Contents inside

the Windows Azure™ group at Microsoft

He is coauthor of Windows Sysinternals Administrator’s Reference, co-creator of the

Sysinternals tools available from Microsoft

TechNet, and coauthor of the Windows Internals

book series

David A Solomon is coauthor of the

Windows Internals book series and has taught

his Windows internals class to thousands of developers and IT professionals worldwide,

including Microsoft staff He is a regular speaker

at Microsoft conferences, including TechNet and PDC

Alex Ionescu is a chief software architect and

consultant expert in low-level system software, kernel development, security training, and

reverse engineering He teaches Windows internals courses with David Solomon, and is

active in the security research community

The definitive guide—fully updated for Windows 7

and Windows Server 2008 R2

Delve inside Windows architecture and internals—and see how core

components work behind the scenes Led by a team of internationally

renowned internals experts, this classic guide has been fully updated

for Windows 7 and Windows Server® 2008 R2—and now presents its

coverage in two volumes

As always, you get critical, insider perspectives on how Windows

operates And through hands-on experiments, you’ll experience its

internal behavior firsthand—knowledge you can apply to improve

application design, debugging, system performance, and support

In Part 2, you will:

• Understand how core system and management mechanisms

work—including object manager, synchronization, Wow64,

Hyper-V®, and the registry

• Examine the data structures and activities behind processes,

threads, and jobs

• Go inside the Windows security model to see how it manages

access, auditing, and authorization

• Explore the Windows networking stack from top to bottom—

including APIs, BranchCache, protocol and NDIS drivers, and

layered services

• Dig into internals hands-on using the kernel debugger,

performance monitor, and other tools

• Focus on fundamental techniques and tools

• Hands-on tutorial with practice files plus eBook

Start Here!

• Beginner-level instruction

• Easy to follow explanations and examples

• Exercises to build your first projects

Alex Ionescu spine = 1.2”

PUBLISHED BY

Microsoft Press

A Division of Microsoft Corporation

One Microsoft Way

Redmond, Washington 98052-6399

Copyright © 2012 by David Solomon and Mark Russinovich

All rights reserved No part of the contents of this book may be reproduced or transmitted in any form or by any means without the written permission of the publisher

Library of Congress Control Number: 2012933511

ISBN: 978-0-7356-6587-3

Printed and bound in the United States of America

First Printing

Microsoft Press books are available through booksellers and distributors worldwide If you need support related

to this book, email Microsoft Press Book Support at mspinput@microsoft.com Please tell us what you think of this book at http://www.microsoft.com/learning/booksurvey

Microsoft and the trademarks listed at http://www.microsoft.com/about/legal/en/us/IntellectualProperty/Trademarks/EN-US.aspx are trademarks of the Microsoft group of companies All other marks are property of their respective owners

The example companies, organizations, products, domain names, email addresses, logos, people, places, and events depicted herein are fictitious No association with any real company, organization, product, domain name, email address, logo, person, place, or event is intended or should be inferred

This book expresses the authors’ views and opinions The information contained in this book is provided without any express, statutory, or implied warranties Neither the authors, Microsoft Corporation, nor its resellers, or distributors will be held liable for any damages caused or alleged to be caused either directly or indirectly by this book

Acquisitions Editor: Devon Musgrave

Developmental Editor: Devon Musgrave

Project Editor: Carol Dillingham

Editorial Production: Curtis Philips

Technical Reviewer: Christophe Nasarre; Technical Review services provided by Content Master,

a member of CM Group, Ltd

Copyeditor: John Pierce

Indexer: Jan Wright

Cover: Twist Creative • Seattle

To our parents, who guided and inspired us to follow our dreams

Contents at a Glance

Windows Internals, Sixth Edition, Part 1 (available separately)

CHAPTER 1 Concepts and Tools

CHAPTER 2 System Architecture

CHAPTER 3 System Mechanisms

CHAPTER 4 Management Mechanisms

CHAPTER 5 Processes, Threads, and Jobs

CHAPTER 6 Security

CHAPTER 7 Networking

Windows Internals, Sixth Edition, Part 2

Windows Internals, Sixth Edition, Part 1

(See appendix for Part 1’s table of contents)

Windows Internals, Sixth Edition, Part 2

Introduction xv

Chapter 8 I/O System 1 I/O System Components 1

The I/O Manager 3

Typical I/O Processing 4

Device Drivers 5

Types of Device Drivers 5

Structure of a Driver .12

Driver Objects and Device Objects 14

Opening Devices .19

I/O Processing 25

Types of I/O 25

I/O Request to a Single-Layered Driver 33

I/O Requests to Layered Drivers 40

I/O Cancellation 48

I/O Completion Ports .53

I/O Prioritization 58

Container Notifications 65

Driver Verifier 65

Kernel-Mode Driver Framework (KMDF) 68

Structure and Operation of a KMDF Driver 68

KMDF Data Model 70

KMDF I/O Model 74

What do you think of this book? We want to hear from you!

User-Mode Driver Framework (UMDF) 78

The Plug and Play (PnP) Manager 81

Level of Plug and Play Support 82

Driver Support for Plug and Play .82

Driver Loading, Initialization, and Installation .84

Driver Installation 94

The Power Manager 98

Power Manager Operation 100

Driver Power Operation 101

Driver and Application Control of Device Power 105

Power Availability Requests 105

Processor Power Management (PPM) 108

Conclusion 123

Chapter 9 Storage Management 125 Storage Terminology 125

Disk Devices 126

Rotating Magnetic Disks 126

Solid State Disks 128

Disk Drivers .131

Winload 132

Disk Class, Port, and Miniport Drivers 132

Disk Device Objects 136

Partition Manager 138

Volume Management 138

Basic Disks 139

Dynamic Disks 141

Multipartition Volume Management 147

The Volume Namespace 153

Volume I/O Operations 159

Virtual Disk Service 160

Virtual Hard Disk Support 162

Attaching VHDs 163

Nested File Systems 163

BitLocker Drive Encryption 163

Encryption Keys 165

Trusted Platform Module (TPM) 168

BitLocker Boot Process 170

BitLocker Key Recovery 172

Full-Volume Encryption Driver 173

BitLocker Management 174

BitLocker To Go .175

Volume Shadow Copy Service .177

Shadow Copies 177

VSS Architecture 177

VSS Operation 178

Uses in Windows .181

Conclusion 186

Chapter 10 Memory Management 187 Introduction to the Memory Manager 187

Memory Manager Components 188

Internal Synchronization 189

Examining Memory Usage 190

Services Provided by the Memory Manager 193

Large and Small Pages .193

Reserving and Committing Pages 195

Commit Limit .199

Locking Memory .199

Allocation Granularity 199

Shared Memory and Mapped Files 200

Protecting Memory 203

No Execute Page Protection .204

Copy-on-Write 209

Address Windowing Extensions .210

Kernel-Mode Heaps (System Memory Pools) .212

Pool Sizes 213

Monitoring Pool Usage 215

Look-Aside Lists 219

Heap Manager 220

Types of Heaps 221

Heap Manager Structure .222

Heap Synchronization 223

The Low Fragmentation Heap 223

Heap Security Features 224

Virtual Address Space Layouts 228

x86 Address Space Layouts .229

x86 System Address Space Layout .232

x86 Session Space .233

System Page Table Entries .235

64-Bit Address Space Layouts 237

x64 Virtual Addressing Limitations 240

Dynamic System Virtual Address Space Management 242

System Virtual Address Space Quotas 245

User Address Space Layout .246

Address Translation 251

x86 Virtual Address Translation 252

Translation Look-Aside Buffer 259

Physical Address Extension (PAE) 260

x64 Virtual Address Translation 265

IA64 Virtual Address Translation 266

Page Fault Handling 267

Invalid PTEs 268

Prototype PTEs 269

In-Paging I/O .271

Collided Page Faults .272

Clustered Page Faults .272

Page Files 273

Commit Charge and the System Commit Limit .275

Commit Charge and Page File Size 278

Stacks .279

User Stacks .280

Kernel Stacks 281

DPC Stack .282

Virtual Address Descriptors 282

Process VADs 283

Rotate VADs .284

NUMA 285

Section Objects 286

Driver Verifier 292

Page Frame Number Database 297

Page List Dynamics 300

Page Priority 310

Modified Page Writer 314

PFN Data Structures 315

Physical Memory Limits 320

Windows Client Memory Limits 321

Working Sets 324

Demand Paging 324

Logical Prefetcher .324

Placement Policy .328

Working Set Management 329

Balance Set Manager and Swapper .333

System Working Sets 334

Memory Notification Events .335

Proactive Memory Management (Superfetch) .338

Components 338

Tracing and Logging 341

Scenarios 342

Page Priority and Rebalancing 342

Robust Performance 344

ReadyBoost 346

ReadyDrive .348

Unified Caching 348

Process Reflection 351

Conclusion 354

Chapter 11 Cache Manager 355 Key Features of the Cache Manager 355

Single, Centralized System Cache 356

The Memory Manager .356

Cache Coherency 356

Virtual Block Caching .358

Stream-Based Caching 358

Recoverable File System Support 359

Cache Virtual Memory Management 360

Cache Size .361

Cache Virtual Size 361

Cache Working Set Size .361

Cache Physical Size .363

File System Interfaces 373

Copying to and from the Cache 374

Caching with the Mapping and Pinning Interfaces 374

Caching with the Direct Memory Access Interfaces .375

Fast I/O 375

Read-Ahead and Write-Behind .377

Intelligent Read-Ahead 378

Write-Back Caching and Lazy Writing 379

Write Throttling 388

System Threads 390

Conclusion 390

Chapter 12 File Systems 391 Windows File System Formats .392

CDFS 392

UDF 393

FAT12, FAT16, and FAT32 .393

exFAT .396

NTFS 397

File System Driver Architecture .398

Local FSDs 398

Remote FSDs 400

File System Operation 407

File System Filter Drivers 413

Troubleshooting File System Problems 415

Process Monitor Basic vs Advanced Modes 415

Process Monitor Troubleshooting Techniques 416

Common Log File System 416

NTFS Design Goals and Features 424

High-End File System Requirements 424

Advanced Features of NTFS 426

NTFS File System Driver 439

NTFS On-Disk Structure 442

Volumes 442

Clusters 442

Master File Table .443

File Record Numbers 447

File Records 447

File Names 449

Resident and Nonresident Attributes 453

Data Compression and Sparse Files 456

The Change Journal File 461

Indexing 464

Object IDs 466

Quota Tracking 466

Consolidated Security 467

Reparse Points .469

Transaction Support .469

NTFS Recovery Support 477

Design .478

Metadata Logging 479

Recovery .483

NTFS Bad-Cluster Recovery 487

Self-Healing 490

Encrypting File System Security 491

Encrypting a File for the First Time 494

The Decryption Process .496

Backing Up Encrypted Files 497

Copying Encrypted Files 497

Conclusion 498

Chapter 13 Startup and Shutdown 499 Boot Process .499

BIOS Preboot .499

The BIOS Boot Sector and Bootmgr 502

The UEFI Boot Process 512

Booting from iSCSI 514

Initializing the Kernel and Executive Subsystems 514

Smss, Csrss, and Wininit 522

ReadyBoot 527

Images That Start Automatically 528

Troubleshooting Boot and Startup Problems .529

Last Known Good 530

Safe Mode 530

Windows Recovery Environment (WinRE) 534

Chapter 14 Crash Dump Analysis 547

Why Does Windows Crash? 547

The Blue Screen 548

Causes of Windows Crashes 549

Troubleshooting Crashes 551

Crash Dump Files .553

Crash Dump Generation 559

Windows Error Reporting 561

Online Crash Analysis 563

Basic Crash Dump Analysis 564

Notmyfault 564

Basic Crash Dump Analysis 565

Verbose Analysis 567

Using Crash Troubleshooting Tools 569

Buffer Overruns, Memory Corruption, and Special Pool .569

Code Overwrite and System Code Write Protection 573

Advanced Crash Dump Analysis 574

Stack Trashes 575

Hung or Unresponsive Systems 577

When There Is No Crash Dump 581

Analysis of Common Stop Codes 585

0xD1 - DRIVER_IRQL_NOT_LESS_OR_EQUAL 585

0x8E - KERNEL_MODE_EXCEPTION_NOT_HANDLED 586

0x7F - UNEXPECTED_KERNEL_MODE_TRAP 588

0xC5 - DRIVER_CORRUPTED_EXPOOL 590

Hardware Malfunctions 593

Conclusion 594

Appendix: Contents of Windows Internals, Sixth Edition, Part 1 595 Index 603

What do you think of this book? We want to hear from you!

Microsoft is interested in hearing your feedback so we can continually improve our books and learning resources for you To participate in a brief online survey, please visit:

microsoft.com/learning/booksurvey

Windows Internals, Sixth Edition is intended for advanced computer professionals

(both developers and system administrators) who want to understand how the

core components of the Microsoft Windows 7 and Windows Server 2008 R2 operating

systems work internally With this knowledge, developers can better comprehend the

rationale behind design choices when building applications specific to the Windows

platform Such knowledge can also help developers debug complex problems System

administrators can benefit from this information as well, because understanding how

the operating system works “under the covers” facilitates understanding the

perfor-mance behavior of the system and makes troubleshooting system problems much

easier when things go wrong After reading this book, you should have a better

under-standing of how Windows works and why it behaves as it does

Structure of the Book

For the first time, the book has been divided in two parts This was done to get the

information out more quickly since it takes considerable time to update the book for

each release of Windows

Part 1 begins with two chapters that define key concepts, introduce the tools used in

the book, and describe the overall system architecture and components The next two

chapters present key underlying system and management mechanisms Part 1 wraps

up by covering three core components of the operating system: processes, threads, and

jobs; security; and networking

Part 2 covers the remaining core subsystems: I/O, storage, memory management,

the cache manager, and file systems Part 2 concludes with a description of the startup

and shutdown processes and a description of crash-dump analysis

History of the Book

This is the sixth edition of a book that was originally called Inside Windows NT

(Microsoft Press, 1992), written by Helen Custer (prior to the initial release of Microsoft

Windows NT 3.1) Inside Windows NT was the first book ever published about Windows

NT and provided key insights into the architecture and design of the system Inside

Windows NT, Second Edition (Microsoft Press, 1998) was written by David Solomon It

updated the original book to cover Windows NT 4.0 and had a greatly increased level

of technical depth

Inside Windows 2000, Third Edition (Microsoft Press, 2000) was authored by David

Solomon and Mark Russinovich It added many new topics, such as startup and down, service internals, registry internals, file-system drivers, and networking It also covered kernel changes in Windows 2000, such as the Windows Driver Model (WDM), Plug and Play, power management, Windows Management Instrumentation (WMI),

shut-encryption, the job object, and Terminal Services Windows Internals, Fourth Edition was

the Windows XP and Windows Server 2003 update and added more content focused

on helping IT professionals make use of their knowledge of Windows internals, such as using key tools from Windows Sysinternals (www.microsoft.com/technet/sysinternals)

and analyzing crash dumps Windows Internals, Fifth Edition was the update for

Windows Vista and Windows Server 2008 New content included the image loader, user-mode debugging facility, and Hyper-V

Sixth Edition Changes

This latest edition has been updated to cover the kernel changes made in Windows 7 and Windows Server 2008 R2 Hands-on experiments have been updated to reflect changes in tools

Hands-on Experiments

Even without access to the Windows source code, you can glean much about Windows internals from tools such as the kernel debugger and tools from Sysinternals and Winsider Seminars & Solutions When a tool can be used to expose or demonstrate some aspect of the internal behavior of Windows, the steps for trying the tool yourself are listed in “EXPERIMENT” boxes These appear throughout the book, and we encour-age you to try these as you’re reading—seeing visible proof of how Windows works internally will make much more of an impression on you than just reading about it will

Topics Not Covered

Windows is a large and complex operating system This book doesn’t cover everything

relevant to Windows internals but instead focuses on the base system components For

example, this book doesn’t describe COM+, the Windows distributed object-oriented

programming infrastructure, or the Microsoft NET Framework, the foundation of

man-aged code applications

Because this is an internals book and not a user, programming, or system

administra-tion book, it doesn’t describe how to use, program, or configure Windows

A Warning and a Caveat

Because this book describes undocumented behavior of the internal architecture and

the operation of the Windows operating system (such as internal kernel structures and

functions), this content is subject to change between releases (External interfaces, such

as the Windows API, are not subject to incompatible changes.)

By “subject to change,” we don’t necessarily mean that details described in this book

will change between releases, but you can’t count on them not changing Any

soft-ware that uses these undocumented interfaces might not work on future releases of

Windows Even worse, software that runs in kernel mode (such as device drivers) and

uses these undocumented interfaces might experience a system crash when running on

a newer release of Windows

Acknowledgments

First, thanks to Jamie Hanrahan and Brian Catlin of Azius, LLC for joining us on this

project—the book would not have been finished without their help They did the bulk

of the updates on the “Security” and “Networking” chapters and contributed to the

update of the “Management Mechanisms” and “Processes and Threads” chapters Azius

provides Windows-internals and device-driver training See www.azius.com for more

information

We want to recognize Alex Ionescu, who for this edition is a full coauthor This is a

reflection of Alex’s extensive work on the fifth edition, as well as his continuing work on

this edition

Also thanks to Daniel Pearson, who updated the “Crash Dump Analysis” chapter His many years of dump analysis experience helped to make the information more practical.

Thanks to Eric Traut and Jon DeVaan for continuing to allow David Solomon access

to the Windows source code for his work on this book as well as continued ment of his Windows Internals courses

develop-Three key reviewers were not acknowledged for their review and contributions

to the fifth edition: Arun Kishan, Landy Wang, and Aaron Margosis—thanks again to them! And thanks again to Arun and Landy for their detailed review and helpful input for this edition

This book wouldn’t contain the depth of technical detail or the level of accuracy it has without the review, input, and support of key members of the Microsoft Windows development team Therefore, we want to thank the following people, who provided technical review and input to the book:

“Crash Dump Analysis” chapter

For the “Networking” chapter, a special thanks to Gianluigi Nusca and Tom Jolly, who really went beyond the call of duty: Gianluigi for his extraordinary help with the BranchCache material and the amount of suggestions (and many paragraphs of

material he wrote), and Tom Jolly not only for his own review and suggestions (which

were excellent), but for getting many other developers to assist with the review Here

are all those who reviewed and contributed to the “Networking” chapter:

■ Shiva Kumar Thangapandi

Amos Ortal and Dotan Elharrar were extremely helpful on NAP, and Shiva Kumar

Thangapandi helped extensively with EAP

Thanks to Gerard Murphy for reviewing the shutdown mechanisms in Windows 7

and clarifying Group Policy behaviors

Thanks to Tristan Brown from the Power Management team at Microsoft for

spend-Thanks to Apurva Doshi for sending Alex a detailed document of cache manager changes in Windows 7, which was used to capture some of the new behaviors and changes described in the book.

Thanks to Matthieu Suiche for his kernel symbol file database, which allowed Alex to discover new and removed fields from core kernel data structures and led to the inves-tigations to discover the underlying functionality changes

Thanks to Cenk Ergan, Michel Fortin, and Mehmet Iyigun for their review and input

on the Superfetch details

The detailed checking Christophe Nasarre, overall technical reviewer, performed contributed greatly to the technical accuracy and consistency in the book

We would like to again thank Ilfak Guilfanov of Hex-Rays (www.hex-rays.com) for the IDA Pro Advanced and Hex-Rays licenses they granted to Alex so that he could speed

up his reverse engineering of the Windows kernel

Finally, the authors would like to thank the great staff at Microsoft Press behind turning this book into a reality Devon Musgrave served double duty as acquisitions editor and developmental editor, while Carol Dillingham oversaw the title as its project editor Editorial and production manager Curtis Philips, copy editor John Pierce, proof-reader Andrea Fox, and indexer Jan Wright also contributed to the quality of this book Last but not least, thanks to Ben Ryan, publisher of Microsoft Press, who continues

to believe in the importance of continuing to provide this level of detail about Windows

to their readers!

Errata & Book Support

We’ve made every effort to ensure the accuracy of this book and its companion tent Any errors that have been reported since this book was published are listed on our Microsoft Press site at oreilly.com:

Please note that product support for Microsoft software is not offered through the

addresses above

We Want to Hear from You

At Microsoft Press, your satisfaction is our top priority, and your feedback our most

valuable asset Please tell us what you think of this book at:

http://www.microsoft.com/learning/booksurvey

The survey is short, and we read every one of your comments and ideas Thanks in

advance for your input!

Stay in Touch

Let’s keep the conversation going! We’re on Twitter: http://twitter.com/MicrosoftPress

C H A P T E R 8

I/O System

The Windows I/O system consists of several executive components that together manage

hard-ware devices and provide interfaces to hardhard-ware devices for applications and the system In this

chapter, we’ll first list the design goals of the I/O system, which have influenced its implementation

We’ll then cover the components that make up the I/O system, including the I/O manager, Plug and

Play (PnP) manager, and power manager Then we’ll examine the structure and components of the

I/O system and the various types of device drivers We’ll look at the key data structures that describe

devices, device drivers, and I/O requests, after which we’ll describe the steps necessary to complete

I/O requests as they move through the system Finally, we’ll present the way device detection, driver

installation, and power management work

I/O System Components

The design goals for the Windows I/O system are to provide an abstraction of devices, both hardware

(physical) and software (virtual or logical), to applications with the following features:

■ Uniform security and naming across devices to protect shareable resources (See Chapter 6,

“Security,” in Part 1 for a description of the Windows security model.)

■ High-performance asynchronous packet-based I/O to allow for the implementation of scalable

applications

■ Services that allow drivers to be written in a high-level language and easily ported between

different machine architectures

■ Layering and extensibility to allow for the addition of drivers that transparently modify the

be-havior of other drivers or devices, without requiring any changes to the driver whose bebe-havior

or device is modified

■ Dynamic loading and unloading of device drivers so that drivers can be loaded on demand

and not consume system resources when unneeded

■ Support for Plug and Play, where the system locates and installs drivers for newly detected

hardware, assigns them hardware resources they require, and also allows applications to

dis-■ Support for power management so that the system or individual devices can enter low power states.

■ Support for multiple installable file systems, including FAT, the CD-ROM file system (CDFS), the Universal Disk Format (UDF) file system, and the Windows file system (NTFS) (See Chapter 12,

“File Systems,” for more specific information on file system types and architecture.)

■ Windows Management Instrumentation (WMI) support and diagnosability so that drivers can

be managed and monitored through WMI applications and scripts (WMI is described in ter 4, “Management Mechanisms,” in Part 1.)

Chap-To implement these features the Windows I/O system consists of several executive components as well as device drivers, which are shown in Figure 8-1

■ The I/O manager is the heart of the I/O system It connects applications and system nents to virtual, logical, and physical devices, and it defines the infrastructure that supports device drivers

compo-■ A device driver typically provides an I/O interface for a particular type of device A driver is a software module that interprets high-level commands, such as read or write, and issues low-level, device-specific commands, such as writing to control registers Device drivers receive commands routed to them by the I/O manager that are directed at the devices they manage, and they inform the I/O manager when those commands are complete Device drivers often use the I/O manager to forward I/O commands to other device drivers that share in the imple-mentation of a device’s interface or control

■ The PnP manager works closely with the I/O manager and a type of device driver called a bus

driver to guide the allocation of hardware resources as well as to detect and respond to the

arrival and removal of hardware devices The PnP manager and bus drivers are responsible for loading a device’s driver when the device is detected When a device is added to a system that doesn’t have an appropriate device driver, the executive Plug and Play component calls on the device installation services of a user-mode PnP manager

■ The power manager also works closely with the I/O manager and the PnP manager to guide the system, as well as individual device drivers, through power-state transitions

■ Windows Management Instrumentation support routines, called the Windows Driver Model (WDM) WMI provider, allow device drivers to indirectly act as providers, using the WDM WMI provider as an intermediary to communicate with the WMI service in user mode (For more information on WMI, see the section “Windows Management Instrumentation” in Chapter 4 in Part 1.)

■ The registry serves as a database that stores a description of basic hardware devices attached

to the system as well as driver initialization and configuration settings (See “The Registry” tion in Chapter 4 in Part 1 for more information.)

sec-■ INF files, which are designated by the inf extension, are driver installation files INF files are the link between a particular hardware device and the driver that assumes primary control of

the device They are made up of script-like instructions describing the device they correspond

to, the source and target locations of driver files, required driver-installation registry tions, and driver dependency information Digital signatures that Windows uses to verify that

modifica-a driver file hmodifica-as pmodifica-assed testing by the Microsoft Windows Hmodifica-ardwmodifica-are Qumodifica-ality Lmodifica-abs (WHQL) modifica-are stored in cat files Digital signatures are also used to prevent tampering of the driver or its INF file

■ The hardware abstraction layer (HAL) insulates drivers from the specifics of the processor and interrupt controller by providing APIs that hide differences between platforms In essence, the HAL is the bus driver for all the devices soldered onto the computer’s motherboard that aren’t controlled by other drivers

WindowsservicesApplications

WMIservice PnP managerUser-mode

User mode Kernel mode

.inf files,.cat files,registry

I/Omanager

Powermanager

PnPmanager

WDM WMIroutines

com-FIGURE 8-1 I/O system components

The I/O Manager

The I/O manager is the core of the I/O system because it defines the orderly framework, or model, within which I/O requests are delivered to device drivers The I/O system is packet driven Most I/O re-

The design allows an individual application thread to manage multiple I/O requests concurrently An IRP is a data structure that contains information completely describing an I/O request (You’ll find more information about IRPs in the section “I/O Request Packets” later in the chapter.)

The I/O manager creates an IRP in memory to represent an I/O operation, passing a pointer to the IRP to the correct driver and disposing of the packet when the I/O operation is complete In contrast,

a driver receives an IRP, performs the operation the IRP specifies, and passes the IRP back to the I/O manager, either because the requested I/O operation has been completed, or because it must be passed on to another driver for further processing

In addition to creating and disposing of IRPs, the I/O manager supplies code that is common to different drivers and that the drivers can call to carry out their I/O processing By consolidating com-mon tasks in the I/O manager, individual drivers become simpler and more compact For example, the I/O manager provides a function that allows one driver to call other drivers It also manages buffers for I/O requests, provides timeout support for drivers, and records which installable file systems are loaded into the operating system There are close to one hundred different routines in the I/O man-ager that can be called by device drivers

The I/O manager also provides flexible I/O services that allow environment subsystems, such as Windows and POSIX, to implement their respective I/O functions These services include sophisti-cated services for asynchronous I/O that allow developers to build scalable, high-performance server applications

The uniform, modular interface that drivers present allows the I/O manager to call any driver out requiring any special knowledge of its structure or internal details The operating system treats all I/O requests as if they were directed at a file; the driver converts the requests from requests made to

with-a virtuwith-al file to hwith-ardwwith-are-specific requests Drivers cwith-an with-also cwith-all ewith-ach other (using the I/O mwith-anwith-ager) to achieve layered, independent processing of an I/O request

Besides providing the normal open, close, read, and write functions, the Windows I/O system vides several advanced features, such as asynchronous, direct, buffered, and scatter/gather I/O, which are described in the “Types of I/O” section later in this chapter

pro-Typical I/O Processing

Most I/O operations don’t involve all the components of the I/O system A typical I/O request starts with an application executing an I/O-related function (for example, reading data from a device) that is processed by the I/O manager, one or more device drivers, and the HAL

As just mentioned, in Windows, threads perform I/O on virtual files A virtual file refers to any source or destination for I/O that is treated as if it were a file (such as files, directories, pipes, and mailslots) The operating system abstracts all I/O requests as operations on a virtual file, because the I/O manager has no knowledge of anything but files, therefore making it the responsibility of the driver to translate file-oriented comments (open, close, read, write) into device-specific commands This abstraction thereby generalizes an application’s interface to devices User-mode applications

(whether Windows or POSIX) call documented functions, which in turn call internal I/O system tions to read from a file, write to a file, and perform other operations The I/O manager dynamically directs these virtual file requests to the appropriate device driver Figure 8-2 illustrates the basic structure of a typical I/O request flow.

HAL hardware access routines

Memory-mapped registers and DMA

FIGURE 8-2 The flow of a typical I/O request

In the following sections, we’ll look at these components more closely, covering the various types

of device drivers, how they are structured, how they load and initialize, and how they process I/O requests Then we’ll cover the operation and roles of the PnP manager and the power manager

Device Drivers

To integrate with the I/O manager and other I/O system components, a device driver must conform to implementation guidelines specific to the type of device it manages and the role it plays in managing the device In this section, we’ll look at the types of device drivers Windows supports as well as the internal structure of a device driver

Types of Device Drivers

Windows supports a wide range of device driver types and programming environments Even within a

for which a driver is intended The broadest classification of a driver is whether it is a user-mode or kernel-mode driver Windows supports a couple of types of user-mode drivers:

■ Windows subsystem printer drivers translate device-independent graphics requests to

printer-specific commands These commands are then typically forwarded to a kernel-mode port driver such as the universal serial bus (USB) printer port driver (Usbprint.sys)

■ User-Mode Driver Framework (UMDF) drivers are hardware device drivers that run in user mode They communicate to the kernel-mode UMDF support library through ALPC See the

“User-Mode Driver Framework (UMDF)” section later in this chapter for more information

In this chapter, the focus is on kernel-mode device drivers There are many types of kernel-mode drivers, which can be divided into the following basic categories:

■ File system drivers accept I/O requests to files and satisfy the requests by issuing their own,

more explicit, requests to mass storage or network device drivers

■ Plug and Play drivers work with hardware and integrate with the Windows power manager and

PnP manager They include drivers for mass storage devices, video adapters, input devices, and network adapters

■ Non–Plug and Play drivers, which also include kernel extensions, are drivers or modules that

extend the functionality of the system They do not typically integrate with the PnP or power managers because they typically do not manage an actual piece of hardware Examples include network API and protocol drivers Process Monitor’s driver, described in Chapter 4 in Part 1, is also an example

Within the category of kernel-mode drivers are further classifications based on the driver model that the driver adheres to and its role in servicing device requests

WDM Drivers

WDM drivers are device drivers that adhere to the Windows Driver Model (WDM) WDM includes support for Windows power management, Plug and Play, and WMI, and most Plug and Play drivers adhere to WDM There are three types of WDM drivers:

■ Bus drivers manage a logical or physical bus Examples of buses include PCMCIA, PCI, USB, and

IEEE 1394 A bus driver is responsible for detecting and informing the PnP manager of devices attached to the bus it controls as well as managing the power setting of the bus

■ Function drivers manage a particular type of device Bus drivers present devices to function

drivers via the PnP manager The function driver is the driver that exports the operational interface of the device to the operating system In general, it’s the driver with the most knowl-edge about the operation of the device

■ Filter drivers logically layer either above or below function drivers (these are called tion filters) or above the bus driver (these are called bus filters), augmenting or changing the

func-behavior of a device or another driver For example, a keyboard capture utility could be mented with a keyboard filter driver that layers above the keyboard function driver.

imple-In WDM, no one driver is responsible for controlling all aspects of a particular device The bus driver is responsible for detecting bus membership changes (device addition or removal), assisting the PnP manager in enumerating the devices on the bus, accessing bus-specific configuration registers, and, in some cases, controlling power to devices on the bus The function driver is generally the only driver that accesses the device’s hardware

Layered Drivers

Support for an individual piece of hardware is often divided among several drivers, each ing a part of the functionality required to make the device work properly In addition to WDM bus drivers, function drivers, and filter drivers, hardware support might be split between the following components:

provid-■ Class drivers implement the I/O processing for a particular class of devices, such as disk,

key-board, or CD-ROM, where the hardware interfaces have been standardized, so one driver can serve devices from a wide variety of manufacturers

■ Miniclass drivers implement I/O processing that is vendor-defined for a particular class of

de-vices For example, although there is a standardized battery class driver written by Microsoft, both uninterruptible power supplies (UPS) and laptop batteries have highly specific interfaces that differ wildly between manufacturers, such that a miniclass is required from the vendor Miniclass drivers are essentially kernel-mode DLLs and do not do IRP processing directly—the class driver calls into them, and they import functions from the class driver

■ Port drivers implement the processing of an I/O request specific to a type of I/O port, such as

SATA, and are implemented as kernel-mode libraries of functions rather than actual device drivers Port drivers are almost always written by Microsoft because the interfaces are typically standardized in such a way that different vendors can still share the same port driver However,

in certain cases, third parties may need to write their own for specialized hardware In some cases, the concept of “I/O port” extends to cover logical ports as well For example, NDIS is the network “port” driver, and Dxgport/Videoprt are the DirectX/video “port” drivers

■ Miniport drivers map a generic I/O request to a type of port into an adapter type, such as a

specific network adapter Miniport drivers are actual device drivers that import the functions supplied by a port driver Miniport drivers are written by third parties, and they provide the interface for the port driver Like miniclass drivers, they are kernel-mode DLLs and do not do IRP processing directly

A simplified example for illustrative purposes will help demonstrate how device drivers work at a high level A file system driver accepts a request to write data to a certain location within a particular file It translates the request into a request to write a certain number of bytes to the disk at a par-

Environment subsystem

or DLL

NtWriteFile(file_handle, char_buffer)

System services

User mode Kernel mode

Write data at specifiedbyte offset within a file

Translate file-relative byteoffset into volume-relativebyte offset and call nextdriver (via I/O manager)

Call driver to write data

at volume-relative byte offset

I/O manager File system

driver

Disk driver

2

31

4

Translate volume-relative byteoffset into disk-relative offsetand transfer data

5

FIGURE 8-3 Layering of a file system driver and a disk driver

This figure illustrates the division of labor between two layered drivers The I/O manager receives a write request that is relative to the beginning of a particular file The I/O manager passes the request

to the file system driver, which translates the write operation from a file-relative operation to a ing location (a sector boundary on the disk) and a number of bytes to write The file system driver calls the I/O manager to pass the request to the disk driver, which translates the request to a physical disk location and transfers the data

start-Because all drivers—both device drivers and file system drivers—present the same framework to the operating system, another driver can easily be inserted into the hierarchy without altering the existing drivers or the I/O system For example, several disks can be made to seem like a very large single disk by adding a driver This logical, volume manager driver is located between the file system and the disk drivers, as shown in the conceptual, simplified architectural diagram presented in Figure 8-4 (For the actual storage driver stack diagram, see Figure 9-3 in Chapter 9, “Storage Manage-ment”) Volume manager drivers are described in more detail in Chapter 9

1 2 3

2

31

Volume manager disk driver

Disk driver

I/O manager System services

Environment subsystem

or DLL

User mode Kernel mode

Call driver to write data atvolume-relative byte offset

Translate volume-relativebyte offset into disknumber and offset, and call next driver (via I/O manager)Call next driver to writedata to disk 3 at disk-relative byte offsetTranslate disk-relative byte offset into physicallocation on disk 3 and transfer data

FIGURE 8-4 Adding a layered driver

EXPERIMENT: Viewing the Loaded Driver List

You can see a list of registered drivers by executing the Msinfo32.exe utility from the Run dialog box of the Start menu Select the System Drivers entry under Software Environment to see the list of drivers configured on the system Those that are loaded have the text “Yes” in the Started column, as shown here:

You can also view the list of loaded kernel-mode drivers with Process Explorer from Windows Sysinternals (http://www.microsoft.com/technet/sysinternals) Run Process Explorer, select the System process, and select DLLs from the Lower Pane View menu entry in the View menu:

Process Explorer lists the loaded drivers, their names, version information (including pany and description), and load address (assuming you have configured Process Explorer to display the corresponding columns).

com-Finally, if you’re looking at a crash dump (or live system) with the kernel debugger, you can

get a similar display with the kernel debugger lm kv command:

Structure of a Driver

The I/O system drives the execution of device drivers Device drivers consist of a set of routines that are called to process the various stages of an I/O request Figure 8-5 illustrates the key driver-function routines

Dispatch

routines Start I/Oroutine

Interruptserviceroutine

FIGURE 8-5 Primary device driver routines

■ An initialization routine The I/O manager executes a driver’s initialization routine, which

is set by the WDK to GSDriverEntry, when it loads the driver into the operating system GSDriverEntry initializes the compiler’s protection against stack-overflow errors (called a

cookie) and then calls DriverEntry, which is what the driver writer must implement The routine

fills in system data structures to register the rest of the driver’s routines with the I/O manager and performs any global driver initialization that’s necessary

■ An add-device routine A driver that supports Plug and Play implements an add-device

routine The PnP manager sends a notification to the driver via this routine whenever a device for which the driver is responsible is detected In this routine, a driver typically creates a device object (described later in this chapter) to represent the device

■ A set of dispatch routines Dispatch routines are the main entry points that a device driver

provides Some examples are open, close, read, and write and any other capabilities the device, file system, or network supports When called on to perform an I/O operation, the I/O manager generates an IRP and calls a driver through one of the driver’s dispatch routines

■ A start I/O routine A driver can use a start I/O routine to initiate a data transfer to or from

a device This routine is defined only in drivers that rely on the I/O manager to queue their incoming I/O requests The I/O manager serializes IRPs for a driver by ensuring that the driver processes only one IRP at a time Drivers can process multiple IRPs concurrently, but serializa-tion is usually required for most devices because they cannot concurrently handle multiple I/O requests

■ An interrupt service routine (ISR) When a device interrupts, the kernel’s interrupt

dis-patcher transfers control to this routine In the Windows I/O model, ISRs run at device rupt request level (DIRQL), so they perform as little work as possible to avoid blocking lower IRQL interrupts (See Chapter 3, “System Mechanisms,” in Part 1 for more information on IRQLs.) An ISR usually queues a deferred procedure call (DPC), which runs at a lower IRQL (DPC/dispatch level), to execute the remainder of interrupt processing (Only drivers for interrupt-driven devices have ISRs; a file system driver, for example, doesn’t have one.)

inter-■ An interrupt-servicing DPC routine A DPC routine performs most of the work involved in

handling a device interrupt after the ISR executes The DPC routine executes at a lower IRQL (DPC/dispatch level) than that of the ISR, which runs at device level, to avoid blocking other interrupts A DPC routine initiates I/O completion and starts the next queued I/O operation on

a device

Although the following routines aren’t shown in Figure 8-5, they’re found in many types of device drivers:

■ One or more I/O completion routines A layered driver might have I/O completion

rou-tines that will notify it when a lower-level driver finishes processing an IRP For example, the I/O manager calls a file system driver’s I/O completion routine after a device driver finishes transferring data to or from a file The completion routine notifies the file system driver about the operation’s success, failure, or cancellation, and it allows the file system driver to perform cleanup operations

■ A cancel I/O routine If an I/O operation can be canceled, a driver can define one or more

cancel I/O routines When the driver receives an IRP for an I/O request that can be canceled, it assigns a cancel routine to the IRP, and as the IRP goes through various stages of processing, this routine can change, or outright disappear, if the current operation is not cancellable If a thread that issues an I/O request exits before the request is completed or cancels the opera-

tion (with the CancelIo Windows function, for example), the I/O manager executes the IRP’s

cancel routine if one is assigned to it A cancel routine is responsible for performing whatever steps are necessary to release any resources acquired during the processing that has already taken place for the IRP as well as for completing the IRP with a canceled status

■ Fast dispatch routines Drivers that make use of the cache manager in Windows (see

Chap-ter 11, “Cache Manager,” for more information on the cache manager), such as file system drivers, typically provide these routines to allow the kernel to bypass typical I/O processing when accessing the driver For example, operations such as reading or writing can be quickly performed by accessing the cached data directly, instead of taking the I/O manager’s usual path that generates discrete I/O operations Fast dispatch routines are also used as a mecha-nism for callbacks from the memory manager and cache manager to file system drivers For instance, when creating a section, the memory manager calls back into the file system driver

to acquire the file exclusively

initialization routine (DriverEntry) are usually released in the unload routine A driver can be

loaded and unloaded while the system is running if the driver supports it, but the unload tine will be called only after all file handles to the device are closed

rou-■ A system shutdown notification routine This routine allows driver cleanup on system

shutdown

■ Error-logging routines When unexpected errors occur (for example, when a disk block

goes bad), a driver’s error-logging routines note the occurrence and notify the I/O manager The I/O manager writes this information to an error log file

Note Most kernel-mode device drivers are written in C Starting with the Windows Driver

Kit 8.0, drivers can also be safely written in C++ due to specific support for kernel-mode C++ in the new compilers Use of assembly language is highly discouraged because of the complexity it introduces and its effect of making a driver difficult to port between hard-ware architectures such as the x86, x64, and IA64

Driver Objects and Device Objects

When a thread opens a handle to a file object (described in the “I/O Processing” section later in this chapter), the I/O manager must determine from the file object’s name which driver it should call to process the request Furthermore, the I/O manager must be able to locate this information the next time a thread uses the same file handle The following system objects fill this need:

■ A driver object represents an individual driver in the system The I/O manager obtains the

ad-dress of each of the driver’s dispatch routines (entry points) from the driver object

■ A device object represents a physical or logical device on the system and describes its

charac-teristics, such as the alignment it requires for buffers and the location of its device queue to hold incoming IRPs It is the target for all I/O operations because this object is what the handle communicates with

The I/O manager creates a driver object when a driver is loaded into the system, and it then calls

the driver’s initialization routine (DriverEntry), which fills in the object attributes with the driver’s entry

points

At any time after loading, a driver creates device objects to represent logical or physical devices, or

even a logical interface or endpoint to the driver, by calling IoCreateDevice or IoCreateDevice Secure

However, most Plug and Play drivers create devices with their add-device routine when the PnP ager informs them of the presence of a device for them to manage Non–Plug and Play drivers, on the other hand, usually create device objects when the I/O manager invokes their initialization routine The I/O manager unloads a driver when the driver’s last device object has been deleted and no refer-ences to the driver remain

man-When a driver creates a device object, the driver can optionally assign the device a name A name places the device object in the object manager namespace, and a driver can either explicitly define

a name or let the I/O manager autogenerate one (The object manager namespace is described

in Chapter 3 in Part 1.) By convention, device objects are placed in the \Device directory in the namespace, which is inaccessible by applications using the Windows API

Note Some drivers place device objects in directories other than \Device For example, the

IDE driver creates the device objects that represent IDE ports and channels in the \Device\Ide directory See Chapter 9 for a description of storage architecture, including the way storage drivers use device objects

If a driver needs to make it possible for applications to open the device object, it must create a symbolic link in the \Global?? directory to the device object’s name in the \Device directory (See Chapter 3 in Part 1 for more information on \??.) Non–Plug and Play and file system drivers typically create a symbolic link with a well-known name (for example, \Device\Hardware2) Because well-known names don’t work well in an environment in which hardware appears and disappears dynami-

cally, PnP drivers expose one or more interfaces by calling the IoRegisterDeviceInterface function,

specifying a GUID (globally unique identifier) that represents the type of functionality exposed GUIDs are 128-bit values that you can generate by using a tool called Uuidgen, which is included with the WDK and the Windows SDK Given the range of values that 128 bits represents, it’s statistically almost certain that each GUID that Uuidgen creates will be forever and globally unique

IoRegisterDeviceInterface generates the symbolic link associated with a device instance; however, a

driver must call IoSetDeviceInterfaceState to enable the interface to the device before the I/O

man-ager actually creates the link Drivers usually do this when the PnP manman-ager starts the device by

send-ing the driver a start-device IRP—in this case, IRP_MJ_PNP, IRP_MN_START_DEVICE.

An application wanting to open a device object whose interfaces are represented with a GUID can

call Plug and Play setup functions in user space, such as SetupDiEnumDeviceInterfaces, to enumerate

the interfaces present for a particular GUID and to obtain the names of the symbolic links it can use

to open the device objects For each device reported by SetupDiEnumDeviceInterfaces, an application executes SetupDiGetDeviceInterfaceDetail to obtain additional information about the device, such as its autogenerated name After obtaining a device’s name from SetupDiGetDeviceInterfaceDetail, the application can execute the Windows function CreateFile to open the device and obtain a handle.

EXPERIMENT: Looking at Device Objects

You can use the WinObj tool from Sysinternals or the !object kernel debugger command to view

the device names under \Device in the object manager namespace The following screen shot shows an I/O manager–assigned symbolic link that points to a device object in \Device with an autogenerated name:

When you run the !object kernel debugger command and specify the \Device directory, you

should see output similar to the following:

When you enter the !object command and specify an object manager directory object, the

kernel debugger dumps the contents of the directory according to the way the object manager organizes it internally For fast lookups, a directory stores objects in a hash table based on a hash of the object names, so the output shows the objects stored in each bucket of the direc-tory’s hash table

As Figure 8-6 illustrates, a device object points back to its driver object, which is how the I/O manager knows which driver routine to call when it receives an I/O request It uses the device object

to find the driver object representing the driver that services the device It then indexes into the driver object by using the function code supplied in the original request; each function code corresponds to

a driver entry point (The function codes shown in Figure 8-6 are described in the section “IRP Stack Locations” later in this chapter.)

A driver object often has multiple device objects associated with it The list of device objects sents the physical or logical devices that the driver controls For example, each partition of a hard disk has a separate device object that contains partition-specific information However, the same hard disk driver is used to access all partitions When a driver is unloaded from the system, the I/O manager uses the queue of device objects to determine which devices will be affected by the removal of the driver

Device controlStart I/OUnloadCancel

Deviceobject Deviceobject Deviceobject(Disk) (Disk

partition) partition)(Disk

Devices operated by this driver

.

.

.

.

.

FIGURE 8-6 The driver object

EXPERIMENT: Displaying Driver and Device Objects

You can display driver and device objects with the kernel debugger !drvobj and !devobj

com-mands, respectively In the following example, the driver object for the keyboard class driver is examined, and its lone device object viewed: