Tài liệu Memory Dump Analysis Anthology- P21 docx

One semi-anonym-ous person asked this question on Crash Dump Analysis forum and here is my answer based on my experience in crash dump analysis and kernel level development: "It depen

002dfc38 7d9472d8 00580a9e 00000000 00000000 Button_WndProc

002dfc64 7d9475c3 7dbfa313 00580a9e 00000000 InternalCallWinProc

002dfcdc 7d9477f6 00000000 7dbfa313 00580a9e UserCallWinProcCheckWow

002dfd54 7d947838 00000000 00000000 002dfd90 DispatchMessageWorker

002dfd64 7d956ca0 00000000 00000000 002dfe90 DispatchMessageW

002dfd90 0040568b 00000000 00000000 002dfe90 IsDialogMessageW

002dfda0 004065d8 00000000 00402a07 00000000 IsDialogMessageW

002dfda8 00402a07 00000000 00000000 00000000 PreTranslateInput

002dfdec 00403c68 00000000 00000000 002dfe90 AfxPreTranslateMessage

002dfdfc 00407920 00000000 002dfe90 002dfe6c AfxInternalPumpMessage

002dfe20 004030a1 00000000 00000000 0042ec18 CWnd::RunModalLoop

002dfe6c 0040110d 00000000 0042ec18 0042ec18 CDialog::DoModal

002dff18 004206fb 00000000 00000000 00000000 InitInstance

002dff28 0040e852 00400000 00000000 00000000 AfxWinMain

002dffc0 7d4e992a 00000000 00000000 00000000 tmainCRTStartup

002dfff0 00000000 0040e8bb 00000000 00000000 BaseProcessStart

We can see that most arguments are zeroes Those that are not, either do not

point to valid data or correspond to function return addresses and frame pointers This

can be seen from the raw stack data as well:

We can compare it with the normal full or minidump saved with other /m

op-tions The data zeroed when we use /mr option is shown in bold (module names and

function offsets are removed for visual clarity):

0:000> kvL 100

ChildEBP RetAddr Args to Child

002df868 00403263 00425ae8 00000111 002df8a8 OnBnClickedButton1

002df878 00403470 002dfe90 000003e8 00000000 _AfxDispatchCmdMsg

002df8a8 00402a27 000003e8 00000000 00000000 OnCmdMsg

002df8cc 00408e69 000003e8 00000000 00000000 OnCmdMsg

002dfda0 004065d8 002e77f8 00402a07 002e77f8 IsDialogMessageW

002dfda8 00402a07 002e77f8 002e77f8 00561878 PreTranslateInput

002dfdb8 00408041 002e77f8 002e77f8 002dfe90 PreTranslateMessage

002dfdc8 00403ae3 00561878 002e77f8 002e77f8 WalkPreTranslateTree

002dfddc 00403c1e 002e77f8 00403b29 002e77f8

AfxInternalPreTranslateMessage

002dfde4 00403b29 002e77f8 00403c68 002e77f8 PreTranslateMessage

002dfdec 00403c68 002e77f8 00000000 002dfe90 AfxPreTranslateMessage

002dfdfc 00407920 00000004 002dfe90 002dfe6c AfxInternalPumpMessage

002dfe20 004030a1 00000004 d5b6c023 0042ec18 RunModalLoop

002dfe6c 0040110d d5b6c037 0042ec18 0042ec18 DoModal

002dff18 004206fb 00000ece 00000002 00000001 InitInstance

002dff28 0040e852 00400000 00000000 001d083e AfxWinMain

002dffc0 7d4e992a 00000000 00000000 7efdf000 tmainCRTStartup

002dfff0 00000000 0040e8bb 00000000 000000c8 BaseProcessStart

CRASH DUMPS AND SECURITY

Suppose you work in a banking industry or for any company that has sensitive

information Is it secure to send a crash dump outside for analysis? One

semi-anonym-ous person asked this question on Crash Dump Analysis forum and here is my answer

based on my experience in crash dump analysis and kernel level development:

"It depends on credit card transaction software design and architecture

and what type of memory dump is configured in Control Panel\System\

Advanced\Startup and Recovery applet: Small, Kernel or Complete

Software usually encrypts data before sending it down TCP/IP stack or

other network protocol If a credit card transaction software doesn't have

any kernel space encryption drivers and doesn't rely on any Microsoft or

other third-party encryption API that might send data to kernel,

communicate to KSECDD or to a user-space component like LSASS via LPC/RPC,

we can safely assume that kernel memory dumps will not have unencrypted

data If encryption is done entirely in user space Small memory dump and

Kernel memory dump will only have encrypted fragments Otherwise there is

a probability that BSOD happens just before encryption or after decryption

or when a secure protocol is being handled This exposure can even happen

in Small memory dumps if BSOD happens in the thread that handles sensitive

information in kernel mode

The same applies if software stores credit data on any medium If it

stores only encrypted data and decrypts entirely in user space without any

transition to kernel it should be safe to enable kernel memory dump

If our goal is ultimate security then even Small memory dump (64Kb) should

not be allowed But in reality as we consider probabilities sending a

small memory dump is equivalent to no more than exposing just one credit

card number or just one password

What we must avoid at any cost is to enable complete memory dump option in

Control Panel In this case all credit card transaction software code and

data including file system cache will be exposed

Contrary to complete memory dump, kernel memory dump will not have much

data even if some potion of it is being communicated during the crash

time."

If you are interested too you can participate in that discussion:

http://www.dumpanalysis.org/forum/viewtopic.php?t=56 or see the solution from

WinDbg (page 600)

PART 11: THE ORIGIN OF CRASH DUMPS

JIT SERVICE DEBUGGING

If we have services running under network service account (prior to Vista) and

they crash we can use NTSD from recent Debugging Tools for Windows and -noio switch

as described in the following article:

NTSD as a better Dr Watson http://www.debuginfo.com/articles/ntsdwatson.html

We need to copy the latest ntsd.exe, dbghelp.dll and dbgeng.dll to some folder if

we don’t want to install Debugging Tools for Windows in a production environment

The example of AeDebug key we can use for 64-bit JIT debugging is

C:\ntsd\ntsd -p %ld -e %ld -g -noio -c ".dump /ma /u c:\TEMP\new.dmp; q"

It is always good to double check these settings with TestDefaultDebugger tool

(page 641)

LOCAL CRASH DUMPS IN VISTA

It appears that Microsoft decided to help customers to save full user dumps

locally for later postmortem analysis According to MSDN this can be done with using

LocalDumps registry key starting from Vista SP1 and Windows Server 2008:

http://msdn2.microsoft.com/en-us/library/bb787181.aspx

This is a quote from the article above:

[…] Prior to application termination, the system will check the registry settings to

determine whether a local dump is to be collected The registry settings control whether

a full dump is collected versus a minidump The custom flags specified also determine

which information is collected in the dump […] You can make use of the local dump

collection even if WER is disabled The local dumps are collected even if the user cancels

WER reporting at any point […]

From my understanding it is independent from the default postmortem debugger

mechanism via AeDebug registry key If it works then full user dump collection might be

easier in production environments because of no need to install Debugging Tools for

Windows to set up a postmortem debugger

COM+ CRASH DUMPS

If we have problems with COM+ components we can configure Component

Ser-vices in Control Panel to save a crash dump:

Refer to the following article for details:

http://msdn.microsoft.com/msdnmag/issues/01/08/ComXP/

If we want to use userdump.exe to save a crash dump when a failing COM+

application displays an error dialog box the following article might help:

http://support.microsoft.com/kb/287643

If we want crash dumps to be automatically collected after some timeout value

refer to the following article for details:

http://support.microsoft.com/kb/910904/

If we have an exception the following article describes how to get a stack trace

from a saved process dump:

http://support.microsoft.com/kb/317317

The following article explains how COM+ handles application faults:

Fault Isolation and Failfast Policy

http://msdn2.microsoft.com/en-us/library/ms679253.aspx

Now I show how to get an error message that was written to event log when

COM+ application was terminated due to a different error code than an access violation

If we get a crash dump from COM+ process we need to look at all threads and find the

one that runs through comsvcs.dll (shown in small font for visual clarity):

0:000> ~*kL

6 Id: 8d4.1254 Suspend: 0 Teb: 7ffd9000 Unfrozen

ChildEBP RetAddr Args to Child

0072ee30 7c822124 77e6baa8 00000394 00000000 ntdll!KiFastSystemCallRet

0072ee34 77e6baa8 00000394 00000000 00000000 ntdll!NtWaitForSingleObject+0xc

0072eea4 77e6ba12 00000394 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac

0072fee0 75bf3e19 75bf3e01 0072ff44 7c81a3c5 comsvcs!CApplication::AsyncApplicationLaunch+0×101

0072feec 7c81a3c5 0bcf5cf8 7c889880 0bcf5d50 comsvcs!CApplication::AppLaunchThreadProc+0×18

FF_DumpProcess function is an indication that the process was being

dumped There is no ComSvcsExceptionFilter function on the thread stack but we can

still get an error message if we look at FailFastStr function arguments:

75c6c740 ―tion Name: My Server‖

75c6c780 ― The serious nature of this err‖

75c6c7c0 ―or has caused the process to ter‖

75c6c800 ―minate…Error Code = 0×80131600″

75c6c840 ‖ : COM+ Services Internals Inf‖

75c6c880 ―ormation: File: d:\nt\com\compl‖

75c6c8c0 ―us\src\comsvcs\srgtapi\csrgtserv‖

75c6c900 ―.cpp, Line: 322 Comsvcs.dll fil‖

75c6c940 ―e version: ENU 2001.12.4720.2517″

75c6c980 ‖ shp‖

Also if we examine parameters of FF_RunCmd call we would see what

applica-tion was used to dump the process:

ChildEBP RetAddr Args to Child

75b8e810 ―RunDll32 comsvcs.dll,MiniDump‖

We can guess that the first parameter is a format string, the second one is a

com-mand line for a process dumper, the third one is PID and the fourth one should be the

name of a dump file to save We can double check this from the raw stack:

ChildEBP RetAddr Args to Child

We can actually find the formatted command that was passed to CreateProcess

call on the raw stack:

CORRECTING MICROSOFT ARTICLE ABOUT USERDUMP.EXE

There is much confusion among Microsoft and Citrix customers on how to use

userdump.exe to save a process dump Microsoft published an article about this tool

and it has the following title:

How to use the Userdump.exe tool to create a dump file:

http://support.microsoft.com/kb/241215/

Unfortunately all scenarios listed there start with:

1 Run the Setup.exe program for your processor

It also says:

<…> move to the version of Userdump.exe for your processor at the command

prompt

I would like to correct the article here We don’t need to run setup.exe, we just

need to copy userdump.exe and dbghelp.dll The latter is important because the version

of that DLL in system32 folder can be older and userdump.exe will not start:

For most customers running setup.exe and configuring the default rules in

Excep-tion Monitor creates the significant amount of False Positive Dumps (page 259) If we

want to manually dump a process we don’t need automatically generated memory

dumps or fine tune Exception Monitor rules to reduce the number of dump files

Just an additional note: if we have an error dialog box showing that a program

got an exception we can find that process in Task Manager and use userdump.exe to

save that process dump manually Then inside the dump it is possible to see that error

Therefore in the case when a default postmortem debugger wasn’t configured in the

registry we can still get a memory dump for postmortem crash dump analysis

Here is an example I removed a postmortem debugger from

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

Debugger=

Now if we run TestDefaultDebugger tool and hit the big crash button we get the

following message box:

If we save TestDefaultDebugger process dump manually using userdump.exe

when this message box is shown:

C:\kktools\userdump8.1\x64>userdump.exe 5264 c:\tdd.dmp

User Mode Process Dumper (Version 8.1.2929.4)

Copyright (c) Microsoft Corp All rights reserved

Dumping process 5264 (TestDefaultDebugger64.exe) to

c:\tdd.dmp

The process was dumped successfully

and open it in WinDbg we can see the problem thread there (shown in small font for

03 00000000`0012e320 00000000`78ee4bdd TestDefaultDebugger64! C_specific_handler+0xa6

04 00000000`0012e3b0 00000000`78ee685a ntdll!RtlpExecuteHandlerForException+0xd

05 00000000`0012e3e0 00000000`78ef3a5d ntdll!RtlDispatchException+0x1b4

The second parameter to RtlDispatchException is the pointer to the exception

context so if we dump the stack trace verbosely we can get that pointer and pass it to

We see that it was NULL pointer dereference that caused the process

termina-tion Now we can dump the full stack trace that led to our crash (shown in small font for

15 00000000`0012fb30 00000000`00404857

TestDefaultDebugger64!AfxInternalPreTranslateMessage+0x64233]

16 00000000`0012fb70 00000000`00404a17 TestDefaultDebugger64!AfxPreTranslateMessage+0x23

17 00000000`0012fba0 00000000`00404a57 TestDefaultDebugger64!AfxInternalPumpMessage+0x37

18 00000000`0012fbe0 00000000`0040a419 TestDefaultDebugger64!AfxPumpMessage+0x1b

The same technique can be used to dump a process when any kind of error

sage box appears, for example, when a NET application displays a NET exception

mes-sage box or a native application shows a run-time error dialog box

WHERE DID THE CRASH DUMP COME FROM?

If our customer complains that the fix we sent yesterday doesn’t work we can

check the computer name from the dump It could be the case that our fix wasn’t

applied to all computers Here is a short summary for different dump types:

1 Complete/kernel memory dumps: dS srv!srvcomputername

dS command shown above interpret the address as a pointer to

UNICODE_STRING structure widely used inside Windows kernel space

Let’s dd the name:

1: kd> dd srv!srvcomputername l2

f5e8d1a0 0022001a e17c9078

Such combination of short integers following by an address is usually an

indica-tion that we have a UNICODE_STRING structure:

CUSTOM POSTMORTEM DEBUGGERS IN VISTA

On the new Vista installation we have neither drwtsn32.exe nor NTSD

Despite that, any application that can attach to a process based on its PID and

save its memory state in a dump file will do as a postmortem debugger The first

ob-vious candidate is userdump.exe which actually can properly setup itself in the registry

Here are the detailed instructions If we already have the latest version of userdump.exe

we can skip the first two steps:

1 Download the latest User Mode Process Dumper from Microsoft At the time

of this writing it has version 8.1

2 Run the downloaded executable file and it will prompt to unzip By default the

current version unzips to c:\kktools\userdump8.1 Do not run setup afterwards because

it is not needed for our purposes

3 Create kktools folder in system32 folder

4 Create the folder where userdump will save our dump files I use

c:\UserDumps in my example

5 Copy dbghelp.dll and userdump.exe from x86 or x64 folder depending on the

version of Windows we use to system32\kktools folder created in step 3

6 Run the elevated command prompt and enter the following command:

C:\Windows\System32\kktools>userdump -I -d c:\UserDumps

User Mode Process Dumper (Version 8.1.2929.5)

Copyright (c) Microsoft Corp All rights reserved

Userdump set up Aedebug registry key

7 Check the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

Debugger=C:\Windows\system32\kktools\userdump -E %ld %ld -D c:\UserDumps\

Auto=0