Tài liệu Memory Dump Analysis Anthology- P8 ppt

We can also do the other way around to extract information from old Windows 2000 crash dumps using WinDbg extensions written for Windows XP and later.. WinDbg !stacks command shows the

WinDbg Tips and Tricks 211

0: kd> ub b8d1a068-2

olddriver!TraceRoutine+0xc1

b8d1a051 mov esp,ebp

b8d1a053 pop ebp

Time: Sat Mar 28 05:48:52 1987

Float: low 1.99384e-019 high 0

Double: 2.68727e-315

OLD DUMPS, NEW EXTENSIONS

Sometimes we can use old Windows 2000 WinDbg extensions to extract

informa-tion from Windows 2003 and XP crash dumps when their native extensions fail We can

also do the other way around to extract information from old Windows 2000 crash

dumps using WinDbg extensions written for Windows XP and later Here is an example

WinDbg !stacks command shows the following not really helpful output from Windows

2000 complete memory dump:

Windows XP and later Kdexts.dll

and we can try newer kdexts.dll with better results:

WinDbg Tips and Tricks 213

8.0002dc 89620680 000000e Blocked cpqasm2+0x5523

8.0002e0 89620400 00000d2 Blocked cpqasm2+0x584d

8.0004ac 895ae9c0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7

8.0004c0 8937b4e0 0018fea Blocked srv!SrvOemStringTo8dot3+0xb7

8.0004a0 895b09e0 0018fe9 Blocked srv!SrvOemStringTo8dot3+0xb7

8.0004cc 893784e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7

8.0004d0 893774e0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7

8.0004d4 893764e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7

OBJECT NAMES AND WAITING THREADS

Sometimes we have threads waiting for synchronization objects like events and it

is good to know their names or vice versa because it might give some clues to whether

the particular thread and object are relevant for the problem For example, we have a

thread from !process 0 ff WinDbg command applied to a complete memory dump:

THREAD 86047968 Cid 01e8.04d4 Teb: 7ffaa000 Win32Thread: 00000000 WAIT:

(Unknown) UserMode Non-Alertable

8604b750 NotificationEvent

86013070 NotificationEvent

Not impersonating

DeviceMap e1007d00

Owning Process 86014ba0 Image: winlogon.exe

Wait Start TickCount 997 Ticks: 788709 (0:03:25:23.578)

Context Switch Count 1

UserTime 00:00:00.000

KernelTime 00:00:00.000

Win32 Start Address USERENV!NotificationThread (0×76929dd9)

Start Address kernel32!BaseThreadStartThunk (0×77e617ec)

Stack Init f5d48000 Current f5d47914 Base f5d48000 Limit f5d45000 Call 0

Priority 10 BasePriority 10 PriorityDecrement 0

Kernel stack not resident

WinDbg Tips and Tricks 215

kd> kv

*** Stack trace for last set context - thread/.cxr resets it

ChildEBP RetAddr Args to Child

Object: 8604b750 Type: (86598990) Event

ObjectHeader: 8604b738 (old version)

HandleCount: 1 PointerCount: 2

kd> !object 86013070

Object: 86013070 Type: (86598990) Event

ObjectHeader: 86013058 (old version)

HandleCount: 10 PointerCount: 18

Directory Object: e19b61c0 Name: userenv: Machine Group Policy has

been applied

We see that one object is named and related to group policies The same

tech-nique can be applied in reverse For example, we want to find which thread is waiting

for 85efb848 event:

kd> !object \BaseNamedObjects

Object: e19b61c0 Type: (865cab50) Directory

ObjectHeader: e19b61a8 (old version)

HandleCount: 75 PointerCount: 259

Directory Object: e10012c8 Name: BaseNamedObjects

Hash Address Type Name

THREAD 8633bd40 Cid 0664.0680 Teb: 7ffde000 Win32Thread: 00000000 WAIT:

(Unknown) UserMode Alertable

85efb848 SynchronizationEvent

8633bdb8 NotificationTimer

Not impersonating

DeviceMap e1007d00

Owning Process 862fa938 Image: VMwareService.exe

Wait Start TickCount 789703 Ticks: 3 (0:00:00:00.046)

Context Switch Count 120485

UserTime 00:00:00.093

KernelTime 00:00:00.062

Win32 Start Address ADVAPI32!ScSvcctrlThreadA (0×77f65e70)

Start Address kernel32!BaseThreadStartThunk (0×77e617ec)

Stack Init f5cc8000 Current f5cc7914 Base f5cc8000 Limit f5cc5000 Call 0

Priority 15 BasePriority 15 PriorityDecrement 0

00a5fe50 77e6202c ntdll!NtWaitForMultipleObjects+0xc

00a5fef8 0040158e kernel32!WaitForMultipleObjectsEx+0×11a

WARNING: Stack unwind information not available Following frames may be

wrong

00a5ff18 00402390 VMwareService+0×158e

00a5ff84 00402f5a VMwareService+0×2390

00a5ffa4 77f65e91 VMwareService+0×2f5a

00a5ffb8 77e64829 ADVAPI32!ScSvcctrlThreadW+0×21

00a5ffec 00000000 kernel32!BaseThreadStart+0×34

WinDbg Tips and Tricks 217

!object command is equivalent to WinObj tool

(http://technet.microsoft.com/en-us/sysinternals/bb896657.aspx) and allows inspecting Windows Object Manager

names-pace that existed at the time when a memory dump was saved Here is the root

direc-tory from my x64 Vista workstation:

lkd> !object \

Object: fffff880000056c0 Type: (fffffa800183fde0) Directory

ObjectHeader: fffff88000005690 (old version)

HandleCount: 0 PointerCount: 50

Directory Object: 00000000 Name: \

Hash Address Type Name

-

01 fffff88000005510 Directory ObjectTypes

03 fffffa80047574e0 Event NETLOGON_SERVICE_STARTED

05 fffff8800156fb00 SymbolicLink SystemRoot

06 fffff880018bfeb0 Directory Sessions

07 fffffa800448eb90 ALPC Port MmcssApiPort

08 fffff8800000a060 Directory ArcName

09 fffff88000081e10 Directory NLS

fffffa80047523c0 ALPC Port XactSrvLpcPort

10 fffffa8004504e60 ALPC Port ThemeApiPort

fffff880018efce0 Directory Windows

fffff88000007bd0 Directory GLOBAL??

fffffa8004199de0 Event LanmanServerAnnounceEvent

fffffa80043027d0 Event DSYSDBG.Debug.Trace.Memory.2a4

11 fffff8800189feb0 Directory RPC Control

13 fffffa8003ed6490 Event EFSInitEvent

14 fffffa8002746bd0 Device clfs

fffff88000fb6b10

15 fffffa8003dd5060 ALPC Port SeRmCommandPort

fffffa80040c7210 Event CsrSbSyncEvent

16 fffff880000052e0 SymbolicLink DosDevices

fffffa8004626c70 Device Cdfs

17 fffff8800471c210 Directory KnownDlls32

fffffa8004770490 ALPC Port AELPort

fffffa8004342680 Event EFSSrvInitEvent

18 fffff8800000a2b0 Key \REGISTRY

fffffa8004851900 ALPC Port WindowsErrorReportingServicePort

19 fffff88004732380 Directory BaseNamedObjects

21 fffff88000072d00 Directory UMDFCommunicationPorts

fffffa8004182120 ALPC Port SmSsWinStationApiPort

fffffa8003ddbe60 Event UniqueInteractiveSessionIdEvent

22 fffff88000875a00 Directory KnownDlls

fffffa8003ece330 Device FatCdrom

fffffa8003a16720 Device Fat

29 fffffa8004574e60 ALPC Port UxSmsApiPort

30 fffff88000013060 Directory Device

fffffa8004342700 Event EFSSmbInitEvent

32 fffffa8004342260 ALPC Port LsaAuthenticationPort

34 fffffa8003dd7e60 ALPC Port SmApiPort

fffff88004bf5080 Section LsaPerformance

fffffa8003f65160 Event UniqueSessionIdEvent

36 fffff88000081c60 Directory Driver

fffffa8004308c00 Event SAM_SERVICE_STARTED

We can inspect any directory or object, for example:

lkd> !object \FileSystem

Object: fffff88000081ab0 Type: (fffffa800183fde0) Directory

ObjectHeader: fffff88000081a80 (old version)

HandleCount: 0 PointerCount: 31

Directory Object: fffff880000056c0 Name: FileSystem

Hash Address Type Name

fffffa8003c6e470 Device CdfsRecognizer

12 fffffa800261c300 Device UdfsDiskRecognizer

fffffa8003c6e680 Driver Fs_Rec

fffffa800468cc90 Driver MRxDAV

fffff88000072eb0 Directory Filters

21 fffffa80046be400 Driver bowser

fffffa8001c92c40 Driver FltMgr

22 fffffa800261cc40 Device FatCdRomRecognizer

23 fffffa8002756e70 Driver Ntfs

24 fffffa8003dc0530 Driver Npfs

fffffa80027abd20 Driver Mup

fffffa80018476a0 Driver RAW

27 fffffa8003f04270 Driver fastfat

28 fffffa8002745060 Driver FileInfo

31 fffffa800261ce50 Device FatDiskRecognizer

33 fffffa80046c4650 Driver srv2

fffffa8003eaf470 Driver NetBIOS

fffffa800261ca30 Device ExFatRecognizer

34 fffffa8003ce3610 Driver SRTSP

35 fffffa800261c060 Device UdfsCdRomRecognizer

WinDbg Tips and Tricks 219

MEMORY DUMPS FROM VIRTUAL IMAGES

Although I haven’t found the way to distinguish the process dump taken from a

physical machine versus virtualized machine there is a way to see it from kernel and

complete memory dumps if VMware Tools are installed inside the guest Windows OS:

In case of a kernel minidump we can check for VMware drivers (as we can

ob-viously do with kernel and complete memory dumps):

kd> lmt m vm*

start end module name

bf9e6000 bf9faa80 vmx_fb Tue Oct 04 08:13:32 2005

f6e8b000 f6e8ed80 vmx_svga Tue Oct 04 08:13:02 2005

f77e7000 f77ede80 vmxnet Sat Apr 22 23:13:11 2006

f7997000 f7998200 vmmouse Tue Aug 02 20:07:49 2005

f79c9000 f79ca5c0 vmmemctl Thu Jul 26 21:50:03 2007

If VMware Tools are not installed we can check machine id:

kd> !sysinfo machineid

Machine ID Information [From Smbios 2.31, DMIVersion 0, Size=1642]

BiosVendor = Phoenix Technologies LTD

BiosVersion = 6.00

BiosReleaseDate = 04/17/2006

SystemManufacturer = VMware, Inc

SystemProductName = VMware Virtual Platform

SystemVersion = None

BaseBoardManufacturer = Intel Corporation

BaseBoardProduct = 440BX Desktop Reference Platform

BaseBoardVersion = None

FILTERING PROCESSES

When I analyze memory dumps coming from Microsoft or Citrix terminal service

environments I frequently need to find a process hosting terminal service In Windows

2000 it was the separate process termsrv.exe and now it is termsrv.dll which can be

loaded into any of several instances of svchost.exe The simplest way to narrow down

that svchost.exe process if we have a complete memory dump is to use the module

op-tion of WinDbg !process command:

!process /m termsrv.dll 0

!process /m wsxica.dll 0

!process /m ctxrdpwsx.dll 0

Note: this option works only with W2K3, XP and later OS

Also to list all processes with user space stacks having the same image name we

can use the following command:

WinDbg Scripts 221

WINDBG SCRIPTS

FIRST ENCOUNTERS

Sometimes instead of writing a debugging extension it is much faster to write a

script After spending some time I wrote the final version of my first script (based on

WinDbg help sample) which can enumerate processes in a complete memory dump and

output their command line

I saved the script below in a text file and used the following command to run it

from WinDbg command prompt: $$><script.txt

$$ WinDbg script to get process command line for all processes in complete

YET ANOTHER WINDBG SCRIPT

One day I got a Windows 2000 server crash dump with 30 IE processes running

and I wanted to find the only one waiting for a specific function I knew there was one

and I wrote the following script to list all processes and their stacks (of course, I already

opened a log in WinDbg to save that huge amount of output):

In memory dumps coming from XP/W2K3 and higher systems you can get all of

this plus PEB and module information for all processes by using !process 0 ff WinDbg

command The command and flags sets process context for every process and reloads

user symbols accordingly

Another alternative would be to use the following command instead of the script:

!for_each_process ".process /r /p @#Process; !process @#Process"

WinDbg Scripts 223

DEADLOCKS AND CRITICAL SECTIONS

The following script will uncover deadlocks and critical section contention in user

mode processes (including services) if we run it against complete memory dump:

Another alternative would be to use the following command instead of the script:

!for_each_process ".process /r /p @#Process; !ntsdexts.locks"

SECURITY PROBLEM

Crash dumps may expose confidential information stored in memory (see Crash

Dumps and Security, page 604) It seems a solution exists which allows to do some sort

of crash dump analysis or at least to identify problem components without sending

complete or kernel memory dumps

This solution takes advantage of WinDbg ability to execute scripts of arbitrary

complexity For example, the script that combines together all frequent commands can

be used for identification of potential problems in memory dumps:

List of all processes’ thread stacks, loaded modues and critical sections (for

complete memory dump) Other commands can be added if necessary

How does all this work? A customer has to install Debugging Tools for

Win-dows from Microsoft This can be done on any workstation and not necessarily in a

production environment Then the customer has to run WinDbg.exe with some

parame-ters including path(s) to symbols (-y), a path to a memory dump (-z) and a path to the

script (-c):

C:\Program Files\Debugging Tools for Windows>WinDbg.exe -y

"srv*c:\mss*http://msdl.microsoft.com/download/symbols" -z MEMORY.DMP -c

"$$><c:\WinDbgScripts\Dmp2Txt.txt;q" -Q -QS -QY –QSY

Once WinDbg.exe finishes processing the memory dump (it can run for couple of

hours if we have many processes in our complete memory dump) we can copy the log

file created in “C:\Program Files\Debugging Tools for Windows” folder, archive it and

$$ KeDmp2Txt: End of File

Note: if the dump is LiveKd.exe generated then due to inconsistency scripts may run forever

For XP/W2K3 and higher you can simplify the script at the cost of excluding

proc-ess critical section locks:

or use the following command instead of for loop:

!for_each_process ".process /r /p @#Process; !process @#Process;

!ntsdexts.locks; lmv"

WinDbg Scripts 227

HUNDREDS OF CRASH DUMPS

Suppose we have 100 - 200 user dumps from various user processes in the

sys-tem and we want to quickly check their thread stacks, locks or to see something

suspi-cious related to our product or its environment our customers are complaining about It

is much easier to collect such information into text files and browse them quickly than

open every crash dump in WinDbg We can use a shell script (VBScript) to automate

loading dumps into WinDbg and use WinDbg scripts to run complex commands against

loaded user dumps For example, we can use the following shell script:

'

' UDumps2Txt.vbs

'

Set fso = CreateObject("Scripting.FileSystemObject")

Set Folder = fso.GetFolder(".")

Set Files = Folder.Files

Set WshShell = CreateObject("WScript.Shell")

For Each File In Files

Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for