Tài liệu Memory Dump Analysis Anthology- P10 doc

We can use .cxr command to change thread execution context to what it was at exception time: After changing the context we can see the thread stack prior to that exception: We see tha

HIDDEN EXCEPTION

Another pattern that occurs frequently is called Hidden Exception It manifests

it-self when we run !analyze -v command and we don’t see an exception or we see only a

breakpoint hit In this case manual analysis is required Sometimes this happens because

of another pattern: Multiple Exceptions (page 255) In other cases an exception

hap-pens and it is handled by an exception handler dismissing it and a process continues its

execution slowly accumulating corruption inside its data leading to a new crash or hang

Sometimes we see a process hanging during its termination like the case shown below

We have a process dump with only one thread:

A second parameter to both functions is a pointer to the so called exception

con-text (processor state when an exception occurred) We can use cxr command to change

thread execution context to what it was at exception time:

After changing the context we can see the thread stack prior to that exception:

We see that the exception happened when component2 was searching a Unicode

string for a character (wcschr) Most likely the string was not zero terminated:

To summarize and show the common exception handling path in user space here

is another thread stack taken from a different dump:

When RtlpCoalesceFreeBlocks (this function compacts a heap and it is called

from RtlFreeHeap) does an illegal memory access then this exception is first processed

in kernel and because it happened in user space and mode the execution is transferred

to RtlDispatchException which searches for exception handlers and in this case there is a

default one installed: UnhandledExceptionFilter

If we see this function on call stack we can also manually get an exception

con-text and a thread stack leading to it like in this example below taken from some other

crash dump:

The most likely reason of this crash is an instance of Dynamic Memory

Corrup-tion pattern - heap corrupCorrup-tion (page 257)

We can also search for exception codes like c0000005 using scripts to dump raw

stack data (see pages 231 and 236) For example:

001b:0100e076 891a mov dword ptr [edx],ebx ds:0023:01073bb0=????????

The presence of unloaded fault handling modules can be the sign of hidden

DEADLOCK (CRITICAL SECTIONS)

The next pattern is called Deadlock If you don’t know what “deadlock” is please

read its explanation on page 31 Deadlocks do not only happen with synchronization

primitives like mutexes, events or more complex objects (built upon primitives) like

criti-cal sections or executive resources (ERESOURCE) They can happen from high level or

systems perspective in inter-process or inter-component communication, for example,

mutually waiting on messages: GUI window messages, LPC messages, RPC calls

How can we see deadlocks in memory dumps? Let’s start with user dumps and

critical sections

First I would recommend reading the following excellent MSDN article to

under-stand various members of CRITICAL_SECTION structure:

Break Free of Code Deadlocks in Critical Sections Under Windows

(msdn.microsoft.com/msdnmag/issues/03/12/CriticalSections/default.aspx)

WinDbg !locks command examines process critical section list and displays all

locked critical sections, lock count and thread id of current critical section owner This is

the output from a memory dump of hanging Windows print spooler process

TID#624 owns CritSec 784B0348 and is waiting for CritSec 76AB8070

TID#1c48 owns CritSec 76AB8070 and is waiting for CritSec 784B0348

0:000>~*kv

12 Id: bc0.624 Suspend: 1 Teb: 7ffd3000 Unfrozen

0000024c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb

76ab8000 76a815ef 76ab8070 NTDLL!RtlpWaitForCriticalSection+0×9e

76ab8070 76a844f8 00cd1f38 NTDLL!RtlEnterCriticalSection+0×46

000cc8f8 0288ea30 0288ea38 NTDLL!LdrpLoadDll+0×2e6

000cc8f8 0288ea30 0288ea38 NTDLL!LdrLoadDll+0×17)

01791faa 0176a684 76a5cd34 WIN32SPL!InternalAddPrinterConnection+0×1b4

01791faa 02c0fa00 02c0fabc WIN32SPL!AddPrinterConnectionW+0×15

000d21d0 02c0fe50 000d3210 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×10c

770c9ad0 00076608 770cb6d8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×229

00076608 770cb6d8 0288f9a8 rpcrt4!RecvLotsaCallsWrapper+0×9

00074a50 02c0ffec 77e7438b rpcrt4!BaseCachedThreadRoutine+0×11f

00076e68 770cb6d8 0288f9a8 rpcrt4!ThreadStartRoutine+0×18

770d1c54 00076e68 00000000 KERNEL32!BaseThreadStart+0×52

This analysis looks pretty simple and easy What about kernel and complete

memory dumps? Of course we cannot see user space critical sections in kernel memory

dumps but we can see them in complete memory dumps after switching to the

appropriate process context and using !ntsdexts.locks This can be done via simple

script adapted from debugger.chm (see Deadlocks and Critical Sections section there)

Why it is so easy to see deadlocks when critical sections are involved? This is

be-cause their structures have a member that records their owner So it is very easy to map

them to corresponding threads The same is with kernel ERESOURCE synchronization

objects Other objects do not have an owner, for example, in case of events it is not so

easy to find an owner just by looking at an event object We need to examine thread call

stacks, other structures or have access to source code

There is also !cs WinDbg extension where !cs -l command lists all locked sections

with stack traces and !cs -t shows critical section tree For the latter we need to enable

Application Verifier using gflags.exe or set 0×100 in registry for your image:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows

NT\CurrentVersion\Image File Execution Options

GlobalFlag=0×00000100

Here is another deadlock example in hanging IE process (stack traces are shown

in smaller font for visual clarity):

CritSec shell32!CMountPoint::_csDL+0 at 7cae42d0

0 Id: c068.b7b4 Suspend: 1 Teb: 7ffdd000 Unfrozen

ChildEBP RetAddr Args to Child

0013bd0c 7c827d0b 7c83d236 000001d0 00000000 ntdll!KiFastSystemCallRet

0013bd10 7c83d236 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc

0013bd4c 7c83d281 000001d0 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3

0013bd6c 7c82f20c 7c8877a0 00000000 0013be68 ntdll!RtlEnterCriticalSection+0xa8

0013bda0 7c82f336 00000000 00000000 0013bde8 ntdll!LdrLockLoaderLock+0x133

0013be1c 7c82f2a3 00000001 00000001 00000000 ntdll!LdrGetDllHandleEx+0x94

0013be38 77e65185 00000001 00000000 0013bea0 ntdll!LdrGetDllHandle+0x18

0013be84 77e6528f 0013bea0 00000000 7cae2f60 kernel32!GetModuleHandleForUnicodeString+0x20

0013c648 779cc917 0018e9b0 001e4dc8 04002000 shdocvw!SHGetAttributes+0x53

0013d728 779cd9c8 0013ddac 00193a50 80004005 shdocvw!CNscTree::_OnCDNotify+0x85

0013d754 779cd964 0013ddac 001a06c8 11281f2a shdocvw!CNscTree::_OnNotify+0x2e1

0013d768 779cd8ff 001a06c8 00010090 0000004e shdocvw!CNscTree::OnWinEvent+0x51

0013d798 75eba756 00193a50 00010090 0000004e shdocvw!CNSCBand::OnWinEvent+0x70

0013d7b8 75eba2a2 00193a50 00010090 0000004e browseui!_FwdWinEvent+0x1d

0013d7ec 75eba357 00010090 0000004e 00000064 browseui!CBandSite::_SendToToolband+0x44

0013d818 75ee2a72 0017de98 00010088 00000000 browseui!CBandSite::OnWinEvent+0x143

0013d864 75ee2b32 0017de98 00010088 0000004e browseui!CBrowserBandSite::OnWinEvent+0x14c

0013d890 75ee2a9a 0000004e 00000064 0013ddac browseui!CBaseBar::_CheckForwardWinEvent+0x88

0013d8ac 75ee29dc 0000004e 00000064 0013ddac browseui!CBaseBar::_OnNotify+0x1c

0013d8c8 75ee2965 00010088 0000004e 00000064 browseui!CBaseBar::v_WndProc+0xd4

0013d918 75ee28fa 00010088 0000004e 00000064 browseui!CDockingBar::v_WndProc+0x447

0013d948 75ee2880 00010088 0000004e 00000064 browseui!CBrowserBar::v_WndProc+0x99

0013d96c 7739b6e3 00010088 0000004e 00000064 browseui!CImpWndProc::s_WndProc+0x65

0013d998 7739b874 75ee2841 00010088 0000004e user32!InternalCallWinProc+0x28

0013da10 7739c2d3 00172e34 75ee2841 00010088 user32!UserCallWinProcCheckWow+0x151

0013da4c 7739c337 006172a0 00618f18 00000064 user32!SendMessageWorker+0x4bd

0013da6c 7743b07f 00010088 0000004e 00000064 user32!SendMessageW+0x7f

0013db04 7743b1ef 0013db1c fffffff4 0013ddac comctl32!CCSendNotify+0xc24

0013db40 774a5ab0 00010088 ffffffff fffffff4 comctl32!SendNotifyEx+0x57

0013dbac 774a652d 0001008a 0000004e 00000064 comctl32!CReBar::_WndProc+0x257

0013dbd0 7739b6e3 0001008a 0000004e 00000064 comctl32!CReBar::s_WndProc+0x2c

0013dbfc 7739b874 774a6501 0001008a 0000004e user32!InternalCallWinProc+0x28

0013dc74 7739c2d3 00172e34 774a6501 0001008a user32!UserCallWinProcCheckWow+0x151

0013dcb0 7739c337 00617350 0060a9c0 00000064 user32!SendMessageWorker+0x4bd

0013dcd0 7743b07f 0001008a 0000004e 00000064 user32!SendMessageW+0x7f

0013dd68 7743b10d 001c8900 fffffff4 0013ddac comctl32!CCSendNotify+0xc24

0013dd7c 7748a032 001c8900 00010001 0013ddac comctl32!CICustomDrawNotify+0x2c

0013e070 7748a8bb 001c8900 001d2aa8 01010060 comctl32!TV_DrawItem+0x356

0013e0f4 7748a9ac 00000154 01010060 00000000 comctl32!TV_DrawTree+0x136

0013e158 7745bdd0 001c8900 00000000 0013e21c comctl32!TV_Paint+0x65

0013e1a4 7739b6e3 00010090 0000000f 00000000 comctl32!TV_WndProc+0x6ea

0013e1d0 7739b874 7745b6e6 00010090 0000000f user32!InternalCallWinProc+0x28

0013e248 7739bfce 0015fce4 7745b6e6 00010090 user32!UserCallWinProcCheckWow+0x151

0013e278 7739bf74 7745b6e6 00010090 0000000f user32!CallWindowProcAorW+0x98

0013e298 77431848 7745b6e6 00010090 0000000f user32!CallWindowProcW+0x1b

0013e614 7c828536 0013e62c 00000018 0013e750 user32! fnDWORD+0x24

0013e640 7739cbb2 7739cb75 00010090 0000005e ntdll!KiUserCallbackDispatcher+0x2e

0013e654 77459d14 00010090 00000200 001c8900 user32!NtUserCallHwndLock+0xc

0013e66c 7745bd2d 00000004 016b0055 00000000 comctl32!TV_OnMouseMove+0x62

0013e6bc 7739b6e3 00010090 00000200 00000000 comctl32!TV_WndProc+0x647

0013e6e8 7739b874 7745b6e6 00010090 00000200 user32!InternalCallWinProc+0x28

0013e760 7739bfce 0015fce4 7745b6e6 00010090 user32!UserCallWinProcCheckWow+0x151

0013e790 7739bf74 7745b6e6 00010090 00000200 user32!CallWindowProcAorW+0x98

0013e7b0 77431848 7745b6e6 00010090 00000200 user32!CallWindowProcW+0x1b

0013eaa8 7739ba92 0015fce4 77431d6c 00010090 user32!UserCallWinProcCheckWow+0x151

0013eb10 7739bad0 0013eb50 00000000 0013eb38 user32!DispatchMessageWorker+0x327

0013eb20 75ed1410 0013eb50 00000000 00176388 user32!DispatchMessageW+0xf

0013eb38 75ed14fc 0013eb50 0013ee50 00000000 browseui!TimedDispatchMessage+0x33

0013ed98 75ec1c83 0015f7e8 0013ee50 0015f7e8 browseui!BrowserThreadProc+0x336

0013ee24 75ec61ef 0015f7e8 0015f7e8 00000000 browseui!BrowserProtectedThreadProc+0x44

0013fea8 779ba3a6 0015f7e8 00000001 00000000 browseui!SHOpenFolderWindow+0x22c

0013fec8 0040243d 00152552 00020d02 ffffffff shdocvw!IEWinMain+0x129

0013ff1c 00402744 00400000 00000000 00152552 iexplore!WinMain+0x316

0013ffc0 77e6f23b 00000000 00000000 7ffde000 iexplore!WinMainCRTStartup+0x182

0013fff0 00000000 004025c2 00000000 78746341 kernel32!BaseProcessStart+0x23

1 Id: c068.d71c Suspend: 1 Teb: 7ffdc000 Unfrozen

ChildEBP RetAddr Args to Child

00d4fea0 7c827cfb 7c80e5bb 00000002 00d4fef0 ntdll!KiFastSystemCallRet

00d4fea4 7c80e5bb 00000002 00d4fef0 00000001 ntdll!NtWaitForMultipleObjects+0xc

00d4ff48 7c80e4a2 00000002 00d4ff70 00000000 ntdll!EtwpWaitForMultipleObjectsEx+0xf7

00d4ffb8 77e64829 00000000 00000000 00000000 ntdll!EtwpEventPump+0x27f

00d4ffec 00000000 7c80e1fa 00000000 00000000 kernel32!BaseThreadStart+0x34

2 Id: c068.cba4 Suspend: 1 Teb: 7ffdb000 Unfrozen

ChildEBP RetAddr Args to Child

012bfe18 7c82783b 77c885ac 000001c4 012bff74 ntdll!KiFastSystemCallRet

012bfe1c 77c885ac 000001c4 012bff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc

3 Id: c068.8604 Suspend: 1 Teb: 7ffda000 Unfrozen

ChildEBP RetAddr Args to Child

013bffec 00000000 776b16e4 00171698 00000000 kernel32!BaseThreadStart+0x34

4 Id: c068.d6dc Suspend: 1 Teb: 7ffd9000 Unfrozen

ChildEBP RetAddr Args to Child

016dfd24 7c827cfb 77e6202c 00000005 016dfd74 ntdll!KiFastSystemCallRet

016dfd28 77e6202c 00000005 016dfd74 00000001 ntdll!NtWaitForMultipleObjects+0xc

016dfdd0 7739bbd1 00000005 016dfdf8 00000000 kernel32!WaitForMultipleObjectsEx+0x11a

016dfe2c 7c919b2e 00000004 016dfe54 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141

016dff50 7c8f7ada 77da3f12 00000000 00000000 shell32!CChangeNotify::_MessagePump+0x3b

016dff54 77da3f12 00000000 00000000 00000000 shell32!CChangeNotify::ThreadProc+0x1e

016dffb8 77e64829 00000000 00000000 00000000 shlwapi!WrapperThreadProc+0x94

016dffec 00000000 77da3ea5 0013dea8 00000000 kernel32!BaseThreadStart+0x34

5 Id: c068.caf4 Suspend: 1 Teb: 7ffd8000 Unfrozen

ChildEBP RetAddr Args to Child

01b1fdb4 7c827cfb 77e6202c 00000002 01b1fe04 ntdll!KiFastSystemCallRet

01b1fdb8 77e6202c 00000002 01b1fe04 00000001 ntdll!NtWaitForMultipleObjects+0xc

01b1fe60 7739bbd1 00000002 01b1fe88 00000000 kernel32!WaitForMultipleObjectsEx+0x11a

01b1febc 6c296601 00000001 01b1fef0 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141

6 Id: c068.d624 Suspend: 1 Teb: 7ffd7000 Unfrozen

ChildEBP RetAddr Args to Child

01c9ff9c 7c826f4b 7c83d424 00000001 01c9ffb0 ntdll!KiFastSystemCallRet

01c9ffa0 7c83d424 00000001 01c9ffb0 00000000 ntdll!NtDelayExecution+0xc

01c9ffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpTimerThread+0x47

01c9ffec 00000000 7c83d3dd 00000000 00000000 kernel32!BaseThreadStart+0x34

7 Id: c068.b4e0 Suspend: 1 Teb: 7ffd6000 Unfrozen

ChildEBP RetAddr Args to Child

01d9fd58 7c827d0b 7c83d236 000001d0 00000000 ntdll!KiFastSystemCallRet

01d9fd5c 7c83d236 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc

01d9fd98 7c83d281 000001d0 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3

01d9fdb8 7c839844 7c8877a0 75eb8b7c 75eb0000 ntdll!RtlEnterCriticalSection+0xa8

01d9fec0 77e6b1bb 75eb0000 75eb0000 001e2f98 ntdll!LdrUnloadDll+0x35

01d9fed4 77da4c1c 75eb0000 0020eec8 77da591b kernel32!FreeLibrary+0x41

01d9feec 7c83a827 0020eec8 7c889080 001e4ec0 shlwapi!ExecuteWorkItem+0x28

01d9ff44 7c83aa0b 77da591b 0020eec8 00000000 ntdll!RtlpWorkerCallout+0x71

01d9ff64 7c83aa82 00000000 0020eec8 001e4ec0 ntdll!RtlpExecuteWorkerRequest+0x4f

01d9ff78 7c839f60 7c83a9ca 00000000 0020eec8 ntdll!RtlpApcCallout+0x11

01d9ffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x61

01d9ffec 00000000 7c839efb 00000000 00000000 kernel32!BaseThreadStart+0x34

8 Id: c068.d5a8 Suspend: 1 Teb: 7ffd5000 Unfrozen

ChildEBP RetAddr Args to Child

01fbb6ac 7c911ab4 00000000 00000000 00000000 shell32!SHParseDisplayName+0xa3

01fbb6d0 7c911a6e 01fbbe60 00000000 00000002 shell32!ILCreateFromPathEx+0x3d

01fbb6ec 7c911a4b 01fbbe60 01fbb700 00000000 shell32!SHILCreateFromPath+0x17

01fbb704 7c95e055 01fbbe60 00000104 01fbc0a0 shell32!ILCreateFromPathW+0x18

01fbbb84 7c9ef49d 01fbbe60 00000000 01fbbbac shell32!SHGetFileInfoW+0x117

01fbc06c 01b4d195 01fbc200 00000000 01fbc0a0 shell32!SHGetFileInfoA+0x6a

WARNING: Stack unwind information not available Following frames may be wrong

01fbc0a4 01b54a20 0000073c 02541f28 00000000 issftran!SSCopyFile+0x27ad

00000000 00000000 00000000 00000000 00000000 issftran!DllUnregisterServer+0x70ad

9 Id: c068.d750 Suspend: 1 Teb: 7ffd4000 Unfrozen

ChildEBP RetAddr Args to Child

0228ff7c 7c8277db 71b25914 000004b4 0228ffc0 ntdll!KiFastSystemCallRet

0228ff80 71b25914 000004b4 0228ffc0 0228ffb4 ntdll!ZwRemoveIoCompletion+0xc

0228ffb8 77e64829 71b259de 00000000 00000000 mswsock!SockAsyncThread+0x69

0228ffec 00000000 71b258ab 001fcd20 00000000 kernel32!BaseThreadStart+0x34

0:000> du 7c8d8828

7c8d8828 "EXPLORER.EXE"

0:000> da 01fbc200

01fbc200 "M:\WINDOWS"

CHANGED ENVIRONMENT

Sometimes the change of operating system version or installing an intrusive

product reveals hidden bugs in software that was working perfectly before that

What happens after installing the new software? If we look at the process dump

we would see many DLLs loaded at their specific virtual addresses Here is the output

from lm WinDbg command after attaching to iexplore.exe process running on Windows

71a50000 71a8f000 mswsock

71a90000 71a98000 wshtcpip

71aa0000 71aa8000 WS2HELP

76e80000 76e8e000 rtutils

76e90000 76ea2000 rasman

76eb0000 76edf000 TAPI32

7e1e0000 7e280000 urlmon

7e290000 7e3ff000 SHDOCVW

Installing or upgrading software can change the distribution of loaded DLLs and

their addresses This also happens when we install some monitoring software which

usually injects their DLLs into every process As a result some DLLs might be relocated or

even the new ones appear loaded And this might influence 3rd-party program behavior

therefore exposing its hidden bugs being dormant when executing the process in old

environment I call this pattern Changed Environment

Let’s look at some hypothetical example Suppose our program has the following

Suppose the pointer p is invalid, dangling, its value has been overwritten and this

happened because of some bug Being invalid that pointer can point to a valid memory

location nevertheless and the value it points to most likely is non-zero Therefore the

body of the “if” statement will be executed Suppose it always happens when we run

the program and every time we execute it the value of the pointer happens to be the

same