Tài liệu Memory Dump Analysis Anthology- P15 ppt

This is the number of times a resource was accessed or, in another words, it is the accumulated number of threads waiting for an object: when a thread tries to acquire an object and is

Trang 1

High Contention 421

HIGH CONTENTION

Some Windows synchronization objects like executive resources and critical

sec-tions have a struct member called ContentionCount This is the number of times

a resource was accessed or, in another words, it is the accumulated number of

threads waiting for an object: when a thread tries to acquire an object and is put into a

wait state the count is incremented Hence the name of this pattern: High Contention

Here is an example In a kernel memory dump we have just one

exclu-sively owned lock and it seems that no other threads were blocked by it at the time the

dump was saved However the high contention count reveals CPU spike:

3: kd> !locks

**** DUMP OF ALL RESOURCE OBJECTS ****

KD: Scanning for held locks

Resource @ 0x8abc11f0 Exclusively owned

Contention Count = 19648535

Threads: 896395f8-01<*>

KD: Scanning for held locks…

Resource @ 0x896fab88 Shared 1 owning threads

Threads: 88c78608-01<*>

15464 total locks, 2 locks currently held

Owning Process 8a035020 Image: MyApp.exe

Wait Start TickCount 36969283 Ticks: 0

Context Switch Count 1926423 LargeStack

UserTime 00:00:53.843

KernelTime 00:13:10.703

Win32 Start Address 0×00401478

Start Address 0×77e617f8

Stack Init ba14b000 Current ba14abf8 Base ba14b000 Limit ba146000 Call 0

Priority 11 BasePriority 6 PriorityDecrement 5

Trang 3

Accidental Lock 423

ACCIDENTAL LOCK

When a system is unresponsive or sluggish we usually check _ERESOURCE locks in

kernel or complete memory dumps to see Deadlock (page 323) or High Resource

Contention (page 421) patterns However there is some chance that reported locks are

purely accidental and appear in a crash dump because they just happened at that time

We need to look at Contention Count, Ticks and KernelTime in both blocking and

blocked threads to recognize an Accidental Lock Also the current version of WinDbg

doesn’t distinguish between prolonged and accidental locks when we use !analyze -v -hang command and merely reports some lock chain it finds among equal alternatives

Here is an example The system was reported hang and kernel memory dump

was saved WinDbg analysis command reports one thread blocking 3 other threads and

the driver on top of the blocking thread stack is AVDriver.sys The algorithm WinDbg

uses to point to specific image name is described in Minidump Analysis section (page

43) and in our case it chooses AVDriver:

BLOCKED_THREAD: 8089d8c0

BLOCKING_THREAD: 8aab4700

LOCK_ADDRESS: 8859a570 (!locks 8859a570)

Resource @ 0x8859a570 Exclusively owned

Trang 4

f592fa88 80932e04 nt!ObpLookupObjectName+0×11f

f592fadc 808ea231 nt!ObOpenObjectByName+0xea

**** DUMP OF ALL RESOURCE OBJECTS ****

Resource @ 0x895a62d8 Shared 1 owning threads

Threads: 89570520-01<*>

Trang 6

Resource @ 0x88524490 Shared 1 owning threads

Threads Waiting On Exclusive Access:

8ad78020 887abdb0 88eb39a8 8aa1f668

Resource @ 0x88748c20 Exclusively owned

18911 total locks, 25 locks currently held

We can ignore shared locks and then concentrate on the last 3 exclusively owned

resources It looks suspicious that Contention Count has the same number as the

num-ber of threads waiting on exclusive access (Numnum-berOfExclusiveWaiters) This means

that these resources had never been used before If we dump locks verbosely we would

see that blocked threads had been waiting no more than 2 seconds, for example, for

resource 0×8859a570:

Trang 7

0: kd> !thread 885d0020; !thread 88a7c020; !thread 8aafc7d8

THREAD 885d0020 Cid 0004.1c34 Teb: 00000000 Win32Thread: 00000000 WAIT:

(Unknown) KernelMode Non-Alertable

89908d50 SynchronizationEvent

885d0098 NotificationTimer

Not impersonating

DeviceMap e10022c8

Owning Process 8ad80648 Image: System

Wait Start TickCount 7689055 Ticks: 127 (0:00:00:01.984)

Context Switch Count 248

UserTime 00:00:00.000

KernelTime 00:00:00.000

Start Address srv!WorkerThread (0xf57c3394)

Stack Init b4136000 Current b4135b74 Base b4136000 Limit b4133000 Call 0

THREAD 88a7c020 Cid 0004.3448 Teb: 00000000 Win32Thread: 00000000 WAIT:

88a7c098 NotificationTimer

Not impersonating

DeviceMap e10022c8

UserTime 00:00:00.000

Stack Init b55dd000 Current b55dcb74 Base b55dd000 Limit b55da000 Call 0

Trang 8

b55dcddc 8088d4e2 nt!PspSystemThreadStartup+0×2e

00000000 00000000 nt!KiThreadStartup+0×16

THREAD 8aafc7d8 Cid 0004.058c Teb: 00000000 Win32Thread: 00000000 WAIT:

8aafc850 NotificationTimer

Not impersonating

DeviceMap e10022c8

UserTime 00:00:00.000

Stack Init f592c000 Current f592bb18 Base f592c000 Limit f5929000 Call 0

ChildEBP RetAddr

f592bb30 80832f7a nt!KiSwapContext+0×26

f592bb5c 8082925c nt!KiSwapThread+0×284

f592bba4 8087c1ad nt!KeWaitForSingleObject+0×346

f592bbe0 8087c3a1 nt!ExpWaitForResource+0xd5

Blocking threads themselves are not blocked and active: the number of ticks

passed since their last wait or preemption is 0 This could be a sign of CPU spike pattern

However their accumulated KernelTime is less than a second:

0: kd> !thread 8aad07d8

THREAD 8aad07d8 Cid 0004.0580 Teb: 00000000 Win32Thread: 00000000 WAIT:

Wait Start TickCount 7689182 Ticks: 0

UserTime 00:00:00.000

KernelTime 00:00:00.125

Trang 9

Stack Init f59d8000 Current f59d7680 Base f59d8000 Limit f59d5000 Call 0

0: kd> !thread 8873c8d8

THREAD 8873c8d8 Cid 0004.2898 Teb: 00000000 Win32Thread: 00000000 WAIT:

UserTime 00:00:00.000

KernelTime 00:00:00.031

Stack Init ac320000 Current ac31f680 Base ac320000 Limit ac31d000 Call 0

0: kd> !thread 8aab4700

THREAD 8aab4700 Cid 0004.0588 Teb: 00000000 Win32Thread: 00000000 WAIT:

UserTime 00:00:00.000

KernelTime 00:00:00.765

Stack Init f5930000 Current f592f680 Base f5930000 Limit f592d000 Call 0

Based on this observation we could say that locks were accidental and indeed,

when the problem happened again, the new dump didn’t show them

Trang 10

PASSIVE THREAD (USER SPACE)

When trying to understand why the particular application or service hangs we

look at Stack Trace Collection pattern (page 409) and hope to find some suspicious

threads that are waiting for a response These are active blocked threads Other threads

may appear waiting but they are merely waiting for some notification or data that may

or may not come during their lifetime and, therefore, are normal In other words, they

are passive and hence the name of the pattern Passive Thread Typical examples from

user space include

The main service thread and dispatch threads (when idle)

A thread waiting for file or registry notifications

A generic RPC/LPC/COM thread waiting for messages

Worker threads waiting for a data to appear in a queue

Window message loops (when idle)

Socket and network protocol threads (when idle)

A thread with function names on its stack trace suggesting that it is a

notifica-tion or listener thread

Of course, sometimes these passive threads can be the reason for an application

or service hang, but from my experience, most of the time they are not, unless there are

other threads which they block Let’s now look at example stack traces

NOTE: Generic threads spawned to service various requests and waiting for data

to arrive can be filtered using !uniqstack WinDbg command Conceptually these threads

are part of the so called thread pool software design pattern

LPC/RPC/COM threads waiting for requests:

70 Id: 8f8.1100 Suspend: 1 Teb: 7ff80000 Unfrozen

Trang 11

Passive Thread (User Space) 431

0c01fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc

Trang 12

2 Id: ecc.c94 Suspend: 1 Teb: 7efac000 Unfrozen

Worker threads waiting for data items to process:

43 Id: 8f8.17c0 Suspend: 1 Teb: 7ff8c000 Unfrozen

Trang 13

1 Id: 13c4.350 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen

Child-SP RetAddr Call Site

Idle main service thread and service dispatch threads:

0 Id: 65c.660 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen

1 Id: 65c.664 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen

Idle window message loops:

10 Id: 65c.514 Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen

11 Id: 65c.9bc Suspend: 1 Teb: 000007ff`fffa0000 Unfrozen

Trang 14

13 Id: ecc.b34 Suspend: 1 Teb: 7ef85000 Unfrozen

Idle socket and network protocol threads:

5 Id: ecc.920 Suspend: 1 Teb: 7efa3000 Unfrozen

0412fa34 760f2a78 WININET!ICAsyncThread::SelectThread+0x242

0412fa3c 76ea19f1 WININET!ICAsyncThread::SelectThreadWrapper+0xd

Function names showing passive nature of threads:

8 Id: 65c.b40 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen

00000000`0259fdc8 00000000`76d9d820 ntdll!NtWaitForSingleObject+0xa

00000000`0259fdd0 000007fe`f8258084 kernel32!WaitForSingleObjectEx+0x9c

00000000`0259fe90 000007fe`fee994e7 wsnmp32!thrNotify+0×9c

00000000`0259fef0 000007fe`fee9967d msvcrt!endthreadex+0×47

00000000`0259ff20 00000000`76d9cdcd msvcrt!endthreadex+0×100

00000000`0259ff50 00000000`76eec6e1 kernel32!BaseThreadInitThunk+0xd

00000000`0259ff80 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

12 Id: 65c.908 Suspend: 1 Teb: 000007ff`fff9e000 Unfrozen

00000000`0368fd48 00000000`76d9d820 ntdll!NtWaitForSingleObject+0xa

00000000`0368fd50 000007fe`fa49afd0 kernel32!WaitForSingleObjectEx+0x9c

Trang 15

00000000`0368fe10 00000000`76d9cdcd

FunDisc!CNotificationQueue::ThreadProc+0×2ec

00000000`0368fe70 00000000`76eec6e1 kernel32!BaseThreadInitThunk+0xd

00000000`0368fea0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

13 Id: 65c.904 Suspend: 1 Teb: 000007ff`fff9c000 Unfrozen

00000000`034af9f8 00000000`76d9ed73 ntdll!NtWaitForMultipleObjects+0xa

01e3fec4 77e6711b ntdll!NtWaitForMultipleObjects+0xc

01e3ff6c 77e61075 kernel32!WaitForMultipleObjectsEx+0x11a

01e3ff88 76928415 kernel32!WaitForMultipleObjects+0x18

01e3ffb8 77e66063 userenv!!NotificationThread+0×5f

01e3ffec 00000000 kernel32!BaseThreadStart+0×34

When in doubt it is always a good idea to examine threads in non-hanging

processes to see their normal idle stack traces See Appendix A: Reference Stack Traces

Trang 16

MAIN THREAD

When we look at a thread and it is not in the Passive Thread pattern list (page

430) and it looks more like Blocked Thread (see Volume 2) we may ask whether it is

Main Thread Every process has at least one thread of execution called main or primary

thread Most GUI applications have window message processing loop inside their main

process thread When a memory dump is saved it is most likely that this thread is

blocked waiting for window or user-defined messages to arrive and can be considered

as Passive Thread If we see it blocked on something else waiting for some time we may

consider the application hanging

Here is an example of the normal iexplore.exe thread stack taken from a kernel

THREAD 88ee2b00 Cid 15a8.1654 Teb: 7ffde000 Win32Thread: a2242018 WAIT:

(WrUserRequest) UserMode Non-Alertable

88f82ee0 SynchronizationEvent

Not impersonating

Owning Process 88de4140

Wait Start TickCount 104916 Elapsed Ticks: 0

UserTime 0:00:04.0484

Start Address KERNEL32!BaseProcessStartThunk (0x7c57b70c)

Stack Init be597000 Current be596cc8 Base be597000 Limit be58f000 Call 0

Priority 12 BasePriority 8 PriorityDecrement 0 DecrementCount 0

Trang 17

Main Thread 437

ChildEBP RetAddr

be596ce0 8042d8d7 nt!KiSwapThread+0x1b1

be596d08 a00019c2 nt!KeWaitForSingleObject+0x1a3

be596d44 a00138c5 win32k!xxxSleepThread+0x18a

be596d54 a00138d1 win32k!xxxWaitMessage+0xe

be596d5c 8046b2a9 win32k!NtUserWaitMessage+0xb

be596d5c 77e3c7cd nt!KiSystemService+0xc9

In the same kernel dump there is another iexplore.exe process with the following

main thread stack which had been blocked for 31 seconds:

PROCESS 8811ca00 SessionId: 21 Cid: 4d18 Peb: 7ffdf000 ParentCid:

THREAD 87ce6da0 Cid 4d18.4c68 Teb: 7ffde000 Win32Thread: a22157b8 WAIT:

(Executive) KernelMode Non-Alertable

b5bd6370 NotificationEvent

IRP List:

885d4808: (0006,00dc) Flags: 00000014 Mdl: 00000000

Not impersonating

Owning Process 8811ca00

Wait Start TickCount 102908 Elapsed Ticks: 2008

UserTime 0:00:01.0125

Start Address KERNEL32!BaseProcessStartThunk (0×7c57b70c)

Stack Init b5bd7000 Current b5bd62f4 Base b5bd7000 Limit b5bcf000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 0

Trang 18

2008 Ticks in Standard Time: 31.375s

Main thread need not be a GUI thread Most input console applications have

ReadConsole calls in normal main process thread stack:

0012fe14 765eaf6a kernel32!ReadConsoleA+0×40

0012fe7c 6ec35196 kernel32!ReadFile+0×84

0012fec0 6ec35616 MSVCR80!_read_nolock+0×201

Trang 20

INSUFFICIENT MEMORY (KERNEL POOL)

Although handle leaks may result in insufficient pool memory, many drivers

allo-cate their own private memory and specify a 4-letter ASCII tag, for example, here is

non-paged pool from my x64 Vista workstation (shown in small font for visual clarity):

lkd> !poolused 3

Sorting by NonPaged Pool Consumed

Pool Used:

NonPaged

Tag Allocs Frees Diff Used

EtwB 304 134 170 6550080 Etw Buffer , Binary: nt!etw

File 32630649 32618671 11978 3752928 File objects

Pool 16 11 5 3363472 Pool tables, etc

Ntfr 204791 187152 17639 2258704 ERESOURCE , Binary: ntfs.sys

FMsl 199039 187685 11354 2179968 STREAM_LIST_CTRL structure , Binary: fltmgr.sys

MmCa 250092 240351 9741 2134368 Mm control areas for mapped files , Binary: nt!mm

ViMm 135503 134021 1482 1783824 Video memory manager , Binary: dxgkrnl.sys

Cont 53 12 41 1567664 Contiguous physical memory allocations for device

drivers

Thre 72558 71527 1031 1234064 Thread objects , Binary: nt!ps

VoSm 872 851 21 1220544 Bitmap allocations , Binary: volsnap.sys

NtFs 8122505 8110933 11572 1190960 StrucSup.c , Binary: ntfs.sys

AmlH 1 0 1 1048576 ACPI AMLI Pooltags

SaSc 20281 14820 5461 1048512 UNKNOWN pooltag ‗SaSc‘, please update pooltag.txt

RaRS 1000 0 1000 960000 UNKNOWN pooltag ‗RaRS‘, please update pooltag.txt

…

If the pool tag is unknown the following Microsoft article KB298102 explains how

to locate the corresponding driver We can also use memory search in WinDbg to locate

kernel space addresses and see what modules they correspond to

WinDbg shows the number of failed pool allocations and also shows a message

when pool usage is nearly its maximum Below I put some examples with possible

Page File: \??\C:\pagefile.sys

Current: 3145728 Kb Free Space: 3001132 Kb

Tiêu đề	High contention
Trường học	University of Information Technology
Chuyên ngành	Computer Science
Thể loại	thesis
Năm xuất bản	2023
Thành phố	Ho Chi Minh City

Định dạng
Số trang	30
Dung lượng	659,47 KB