Tài liệu Memory Dump Analysis Anthology- P16 doc

For information please look at the following document: http://www.microsoft.com/whdc/devtools/tools/vistaverifier.mspx Other information that can be included in process, kernel and co

Trang 1

Busy System 451

8088d319 and byte ptr [edi+5Dh],0 8088d31d jmp nt!KiIdleLoop+0xa (8088d262) nt!KiIdleLoop+0xca: 8088d322 cmp byte ptr [ebx+0AA3h],0 8088d329 je nt!KiIdleLoop+0x2 (8088d25a) nt!KiIdleLoop+0xd7: 8088d32f sti 8088d330 lea ecx,[ebx+120h] 8088d336 call nt!KiIdleSchedule (808343e6) 8088d33b test eax,eax 8088d33d mov esi,eax 8088d33f mov edi,dword ptr [ebx+12Ch] 8088d345 jne nt!KiIdleLoop+0x99 (8088d2f1) nt!KiIdleLoop+0xef: 8088d347 jmp nt!KiIdleLoop+0xa (8088d262) In some memory dumps taken when systems or sessions were hanging or very slow for some time we might see Busy System pattern where all processors execute non-idle threads and there are threads in ready queues waiting to be scheduled: 3: kd> !running System Processors f (affinity mask) Idle Processors 0 Prcb Current Next 0 ffdff120 88cef850

1 f7727120 8940b7a0

2 f772f120 8776f020

3 f7737120 87b25360

3: kd> !ready

Processor 0: Ready Threads at priority 8

THREAD 88161668 Cid 3d58.43a0 Teb: 7ffdf000 Win32Thread: bc1eba48

READY

THREAD 882d0020 Cid 1004.0520 Teb: 7ffdf000 Win32Thread: bc230838

READY

THREAD 88716b40 Cid 2034.241c Teb: 7ffdd000 Win32Thread: bc11b388

READY

THREAD 88bf7978 Cid 2444.2564 Teb: 7ffde000 Win32Thread: bc1ccc18

READY

THREAD 876f7a28 Cid 2308.4bfc Teb: 7ffdd000 Win32Thread: bc1f7b98

READY

THREAD 8a3925a8 Cid 0004.0008 Teb: 00000000 Win32Thread: 00000000

READY

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

Trang 2

THREAD 87e69db0 Cid 067c.3930 Teb: 7ffdb000 Win32Thread: bc180990

READY

THREAD 88398c70 Cid 27cc.15b4 Teb: 7ffde000 Win32Thread: bc159ea8

READY

THREAD 8873cdb0 Cid 4c24.4384 Teb: 7ffdd000 Win32Thread: bc1c9838

READY

THREAD 89f331e0 Cid 453c.4c68 Teb: 7ffdf000 Win32Thread: bc21dbd0

READY

THREAD 889a03f0 Cid 339c.2fcc Teb: 7ffdf000 Win32Thread: bc1cdbe8

READY

THREAD 87aacdb0 Cid 3b80.4ed0 Teb: 7ffde000 Win32Thread: bc1c5d10

READY

Processor 3: No threads in READY state

Here is another example from busy 8-processor system where only one processor

was idle at the time of the bugcheck:

5: kd> !ready

5: kd> !running

System Processors ff (affinity mask)

Idle Processors 1

Prcb Current Next

1 f7727120 8713a5a0

2 f772f120 86214750

3 f7737120 86f87020

4 f773f120 86ffe700

5 f7747120 86803a90

6 f774f120 86043db0

7 f7757120 86bcbdb0

5: kd> !thread 8713a5a0 1f

THREAD 8713a5a0 Cid 4ef4.4f04 Teb: 7ffdd000 Win32Thread: bc423920

RUNNING on processor 1

Not impersonating

DeviceMap e44e9a40

Owning Process 864d1d88 Image: SomeExe.exe

Wait Start TickCount 1415535 Ticks: 0

Trang 3

Busy System 453

UserTime 00:06:59.218

KernelTime 00:19:26.359

Win32 Start Address BROWSEUI!BrowserProtectedThreadProc (0x75ec1c3f)

Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

Stack Init b68b8a70 Current b68b8c28 Base b68b9000 Limit b68b1000 Call

b68b8cac bf8a137a win32k!xxxCallHook+0x26

b68b8cec bf85af67 win32k!xxxSendMessageTimeout+0x1e3

Owning Process 85790020 Image: SomeExe.exe

Context Switch Count 1745682 LargeStack

UserTime 00:01:20.031

Win32 Start Address 0x75ec1c3f

Stack Init b4861000 Current b4860558 Base b4861000 Limit b4856000 Call 0

Priority 13 BasePriority 13 PriorityDecrement 0

ChildEBP RetAddr

b4860bd8 bf8da699 nt!PsGetThreadProcess

b4860bf4 bf89d6e6 win32k!IsRestricted+0x2f

b4860c90 bf89da19 win32k!xxxCallHook2+0x12d

b4860cac bf8a137a win32k!xxxCallHook+0x26

b4860cec bf85af67 win32k!xxxSendMessageTimeout+0x1e3

Trang 4

Owning Process 8850e020 Image: lsass.exe

Context Switch Count 39608

UserTime 00:00:01.625

Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c7b0f5)

Stack Init f4925000 Current f4924c38 Base f4925000 Limit f4922000 Call 0

Owning Process 87005708 Image: WINWORD.EXE

UserTime 00:01:00.750

Win32 Start Address WINWORD (0x300019b0)

Start Address kernel32!BaseProcessStartThunk (0x77e617f8)

Stack Init f3465000 Current f3464c48 Base f3465000 Limit f345e000 Call 0

ChildEBP RetAddr

f3464d64 7c8285eb nt!KiFastCallEntry+0x91

f3464d68 badb0d00 ntdll!KiFastSystemCall+0x3

Trang 5

Owning Process 857d5500 Image: SystemDump.exe

UserTime 00:00:00.015

*** ERROR: Module load completed but symbols could not be loaded for

SystemDump.exe

Win32 Start Address SystemDump_400000 (0x0040fe92)

Start Address kernel32!BaseProcessStartThunk (0x77e617f8)

Stack Init b38a4000 Current b38a3c08 Base b38a4000 Limit b389f000 Call 0

ChildEBP RetAddr Args to Child

b38a3bf0 f79e3743 000000e2 cccccccc 866962b0 nt!KeBugCheckEx+0x1b

WARNING: Stack unwind information not available Following frames may be

86dc99a0: (0006,0094) Flags: 00000a00 Mdl: 00000000

Impersonation token: e7b30030 (Level Impersonation)

DeviceMap e4e470a8

Owning Process 891374a8 Image: SomeSvc.exe

Wait Start TickCount 1415215 Ticks: 320 (0:00:00:05.000)

Context Switch Count 11728

UserTime 00:00:02.546

Win32 Start Address 0x0082b983

LPC Server thread working on message Id 82b983

Stack Init b49c1000 Current b49c0a7c Base b49c1000 Limit b49be000 Call 0

ChildEBP RetAddr

b49c0b80 8087c9c0 hal!KeReleaseQueuedSpinLock+0x2d

b49c0ba0 8087ca95 nt!ExReleaseResourceLite+0xac

b49c0ba4 f6faa5ae nt!ExReleaseResourceAndLeaveCriticalRegion+0x5

b49c0bb8 f6faad05 termdd!_IcaCallStack+0x60

Trang 6

Owning Process 872fb708 Image: SomeExe.exe

UserTime 00:10:09.343

Win32 Start Address 0x75ec1c3f

Start Address 0x77e617ec

Stack Init b86cb000 Current b86ca58c Base b86cb000 Limit b86c2000 Call 0

b86caa18 f75fa8c7 nt!IofCallDriver+0x45

b86caa40 f75faa5a SomeFlt!PassThrough+0xbb

b86caa5c 8081df65 SomeFlt!Create+0xda

b86caa70 808f8f71 nt!IofCallDriver+0x45

b86cab58 80937942 nt!IopParseDevice+0xa35

b86cabd8 80933a76 nt!ObpLookupObjectName+0x5b0

b86cac2c 808eae25 nt!ObOpenObjectByName+0xea

b86caca8 808ec0bf nt!IopCreateFile+0x447

b86cad04 808efc4f nt!IoCreateFile+0xa3

b86cad44 8088978c nt!NtOpenFile+0x27

b86cad44 7c8285ec nt!KiFastCallEntry+0xfc

Running threads have good chance to be Spiking Threads (page 305)

Trang 7

Historical Information 457

HISTORICAL INFORMATION

Although crash dumps are static in nature they contain Historical

Informa-tion about past system dynamics that might give clues to a problem and help with

troubleshooting and debugging

For example, IRP flow between user processes and drivers is readily available in

any kernel or complete memory dump WinDbg !irpfind command will show the list

of currently present I/O request packets !irp command will give individual packet

de-tails

Recent Driver Verifier improvements in Vista and Windows Server 2008 allow to

embed stack traces associated with IRP allocation, completion and cancellation For

information please look at the following document:

http://www.microsoft.com/whdc/devtools/tools/vistaverifier.mspx

Other information that can be included in process, kernel and complete memory

dumps may reveal some history of function calls beyond the current snapshot of thread

stacks:

Heap allocation stack traces that are usually used for debugging memory leaks

Handle traces that are used to debug handle leaks (!htrace command)

Raw stack data interpreted symbolically Some examples include dumping

stack data from all process threads and dumping kernel mode stack data

LPC messages (!lpc thread)

Waiting Thread Time pattern (page 343)

Trang 8

IRP DISTRIBUTION ANOMALY

In kernel or complete memory dumps coming from hanging or slow workstations

and servers !irpfind WinDbg command may show IRP Distribution Anomaly pattern

when certain drivers have excessive count of active IRPs not observed under normal

circumstances I created two IRP distribution graphs from two problem kernel dumps by

preprocessing command output using Visual Studio keyboard macros to eliminate

completed IRPs and then using Excel In one case it was a big number of I/O request

packets from 3rd-party antivirus filter driver:

\Driver\3rdPartyAvFilter

In the second case it was the huge number of active IRPs targeted to kernel

socket ancillary function driver:

Trang 9

IRP Distribution Anomaly 459

\Driver\AFD

Two other peaks on both graphs are related to NTPS and NTFS, pipes and file

system and usually normal Here is IRP distribution graph from my Vista workstation

captured while I was writing this post:

Trang 10

LOCAL BUFFER OVERFLOW

Local Buffer Overflow pattern It is observed on x86 platforms when a local

varia-ble and a function return address and/or saved frame pointer EBP are overwritten with

some data As a result, the instruction pointer EIP becomes Wild Pointer (see Volume 2)

and we have a process crash in user mode or a bugcheck in kernel mode Sometimes

this pattern is diagnosed by looking at mismatched EBP and ESP values and in the case

of ASCII or UNICODE buffer overflow EIP register may contain 4-char or 2-wchar_t value

and ESP or EBP or both registers might point at some string fragment like in the example

Trang 11

Passive System Thread (Kernel Space) 461

PASSIVE SYSTEM THREAD (KERNEL SPACE)

Previously we discussed Passive Thread pattern (page 430) in user space Here I

continue with kernel space and passive system threads that don’t run in any user

process context These threads belong to the so called System process, don’t have any

user space stack and their full stack traces can be seen from the output of !process

com-mand (if not completely paged out) or from [System] portion of !stacks 2 comcom-mand:

1: kd> !process 0 ff System

Some system threads from that list belong to core OS functionality and are not

passive (function offsets can vary for different OS versions and service packs):

Trang 12

Other threads belong to various worker queues (they can also be seen from

!exqueue ff command output) and they wait for data items to arrive (passive threads):

Trang 13

Passive System Thread (Kernel Space) 463

Non-Exp system threads having Worker, Logging or Logger substrings in their

function names are passive threads and they wait for data too, for example:

Trang 14

Any deviations in a memory dump can raise suspicion like in the stack below for

Trang 15

Early Crash Dump 465

EARLY CRASH DUMP

Some bugs are fixed using brute-force approach via putting an exception handler

to catch access violations and other exceptions Long time ago I saw one such

“incredi-ble fix” when the image processing application was crashing after approximately

Nth heap free runtime call To ignore crashes a SEH handler was put in place but the

application started to crash in different places Therefore the additional fix was to skip

free calls when approaching N and resume afterwards The application started to crash

less frequently

Here getting Early Crash Dump when a first-chance exception happens can help

in component identification before corruption starts spreading across data Recall that

when an access violation happens in a process thread in user mode the system

gene-rates the first-chance exception which can be caught by an attached debugger and if

there is no such debugger the system tries to find an exception handler and if that

exception handler catches and dismisses the exception the thread resumes its normal

execution path If there are no such handlers found the system generates the so called

second-chance exception with the same exception context to notify the attached

debug-ger and if it is not attached a default thread exception handler usually saves a

postmor-tem user dump

We can get first-chance exception memory dumps with:

Debug Diagnostics

c458-46f1-b24d-f60151d875a3&displaylang=en)

ADPlus in crash mode from Debugging Tools for Windows

Exception Monitor from User Mode Process

Dumper package (http://www.microsoft.com/downloads/details.aspx?FamilyI D=e089ca41-6a87-40c8-bf69-28ac08570b7e&DisplayLang=en)

Here is an example configuration rule for crashes in Debug Diagnostic tool for

TestDefaultDebugger process (Unconfigured First Chance Exceptions option is set to Full

Userdump):

Trang 16

When we push the big crash button in TestDefaultDebugger dialog box two crash

dumps are saved, with the first and second-chance exceptions pointing to the same

code:

Loading Dump File [C:\Program Files (x86)\DebugDiag\Logs\Crash rule for

all instances of TestDefaultDebugger.exe\TestDefaultDebugger PID 4316

Date 11_21_2007 Time_04_28_27PM 2 First chance exception

0XC0000005.dmp]

User Mini Dump File with Full Memory: Only application data is available

Comment: 'Dump created by DbgHost First chance exception 0XC0000005′

Symbol search path is:

srv*c:\mss*http://msdl.microsoft.com/download/symbols

Executable search path is:

Windows Vista Version 6000 MP (2 procs) Free x86 compatible

Product: WinNt, suite: SingleUserTS

Debug session time: Wed Nov 21 16:28:27.000 2007 (GMT+0)

System Uptime: 0 days 23:45:34.711

Process Uptime: 0 days 0:01:09.000

Tiêu đề	Memory Dump Analysis Patterns in Busy System Scenarios
Trường học	University of Technology and Education
Chuyên ngành	Memory Dump Analysis
Thể loại	Thesis

Định dạng
Số trang	30
Dung lượng	771,57 KB