Tài liệu Memory Dump Analysis Anthology- P22 pdf

In summary, on x64 Windows to save a memory dump file of a 64-bit application we can use: x64\userdump.exe \Windows\System32\ntsd.exe 64-bit version of WinDbg.exe to save a memory dum

Dumping Processes Without Breaking Them 631

DUMPING PROCESSES WITHOUT BREAKING THEM

We can do it on any Windows system after Windows 2000 without installing any

additional tools like Userdump or WinDbg And a process won’t be interrupted while its

memory dump is being saved and will continue to work We can use the following

com-mand:

ntsd -pvr -p 'PID' -c ".dump /ma /u process.dmp; q"

PID is a decimal process ID we can get from Task Manager, for example

Note: on x64 system to dump a 32-bit process (shown as *32 in Task Manager)we

need to use NTSD from \Windows\SysWOW64 folder (page 633) On Windows Vista,

NTSD is no longer included but it can be found in Debugging Tools for Windows package

USERDUMP.EXE ON X64

If we install the latest Microsoft user mode process dumper on x64 Windows we

would see both x86 and x64 folders

One advice here: do not dump 32-bit applications and services (shown as *32 in

Task Manager) using userdump.exe from x64 folder: use userdump.exe from x86 folder

32-bit application runs in WOW64 emulation layer on x64 Windows and that emulation

layer is itself native 64-bit process so x64 userdump.exe saves that emulation layer and

not the original 32-bit application If we open that dump file in WinDbg we would see

WOW64 thread stacks and not thread stacks from our original 32-bit application

In summary, on x64 Windows

to save a memory dump file of a 64-bit application we can use:

x64\userdump.exe

\Windows\System32\ntsd.exe

64-bit version of WinDbg.exe

to save a memory dump file of a 32-bit application use:

x86\userdump.exe

\Windows\SysWOW64\ntsd.exe

32-bit WinDbg.exe

NTSD on x64 Windows 633

NTSD ON X64 WINDOWS

If we need to attach NTSD to a process on x64 Windows and to save a memory

dump file we should remember that there are two versions of NTSD: x86 (32-bit) and

x64 The former is located in \Windows\SysWOW64 folder and should be used for

attaching to 32-bit applications and services For explanation why you need different

versions of NTSD please refer to the first picture in Dumps, Debuggers and

Virtualiza-tion (page 516)

If we use WinDbg for that purpose we should install both 32-bit and 64-bit

ver-sions

If we want to install NTSD or WinDbg as a default postmortem debugger we

should use Wow6432Node registry hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows

NT\CurrentVersion\AeDebug

Debugger = ntsd -p %ld -e %ld -g -c ".dump /ma /u c:\TEMP\new.dmp; q"

Please refer to the following Citrix support articles explaining and describing in

more detail how to set NTSD and WinDbg as default postmortem debuggers:

How to Set NTSD as a Default Windows Postmortem Debugger

(http://support.citrix.com/article/CTX105888)

How to Set WinDbg as a Default Windows Postmortem Debugger

(http://support.citrix.com/article/CTX107528)

NEED A DUMP? COMMON USE CASES

The most common scenarios technical support people encounter when facing the

need to create a dump file are:

the article is applicable to any process

Need a system dump from a remote session? Use SystemDump (page 646)

http://support.citrix.com/article/CTX111072

Got correct dump? Use Citrix DumpCheck

http://support.citrix.com/article/CTX108825 (Explorer extension)

http://support.citrix.com/article/CTX108890 (Command line version)

Memory Dump Analysis Using Excel 635 PART 12: TOOLS

MEMORY DUMP ANALYSIS USING EXCEL

Some WinDbg commands output data in tabular format so it is possible to save

their output into a text file, import it to Excel and do sorting, filtering, and graph

visualization Some commands from WinDbg include:

!stacks 1 - Lists all threads with Ticks column so we can sort and filter threads

that had been waiting for no more than 100 ticks, for example

!irpfind - Here we can create various histograms, for example, IRP distribution

based on [Driver] column

The following graph depicts thread distribution in PID - TID coordinates on a busy

multiprocessor system with 25 user sessions and more than 3,000 threads:

WinDbg scripts offer the possibility to output various tabulated data via PRINTF:

0:000> printf "a\tb\tc"

a b c

TESTDEFAULTDEBUGGER.NET

Sometimes there are situations when we need to test exception handling to see

whether it works and how to get dumps or logs from it For example, a customer reports

infrequent process crashes but no crash dumps are saved Then we can try some

application that crashes immediately to see whether it results in error messages and/or

saved crash dumps This was the motivation behind TestDefaultDebugger

pack-age Unfortunately it contains only native applications and we also needed to test NET

CLR exception handling and see what messages it shows in an environment This is a

simple program in C# that creates an empty Stack object and then calls its Pop method

which triggers “Stack empty” exception:

The updated package now includes TestDefaultDebugger.NET.exe and can be

downloaded from Citrix support web site: http://support.citrix.com/article/CTX111901

Cons of Symbol Server 637

CONS OF SYMBOL SERVER

Symbol servers are great However I found that in crash dump analysis the

ab-sence of automatically loaded symbols sometimes helps to identify a problem or at least

gives some directions for further research It also helps to see which hot fixes or service

packs for a product were installed on a problem computer The scenario I use

some-times when I analyze crash dumps from product A is the following:

1 Set up WinDbg to point to Microsoft Symbol Server

2 Load a crash dump and enter various commands based on the issue Some OS

or product A components become visible and their symbols are unresolved

3 From unresolved OS symbols I’m aware of the latest fixes or privates from

Microsoft

4 From unresolved symbols of the product A and PDBFinder tool I determine the

base product level and this already gives me some directions

5 I add the base product A symbols to symbol file path and continue my analysis

6 If unresolved symbols of the product A continue to come up I use PDBFinder

tool again to find corresponding symbols and add them to symbol file path By doing that I’m aware of the product A hot fix and/or service pack level

7 Also from PDBFinder tool I know whether there are any updates to the

component in question

Of course, all of this works only if we store all PDB files from all our fixes and

ser-vice packs in some location(s) with easily identified names or abbreviations, for

example, PRODUCTA\VER20\SP31\FIX01 Adding symbols manually helps to be focused

on components, gives attention to some threads where they appear We might think it

is a waste of time but it only takes very small percentage of time especially if we look at

the memory dump for a couple of hours

What is PDBFinder tool? This is a program I developed to find right symbol files

(especially for minidumps) It scans all locations for PDB or DBG files and adds them to a

binary database Next time we run PDBFinder tool it loads that database and we can

find PDB or DBG file location by specifying module name and its date We can also do a

fuzzy search by specifying some date interval If we run it with -update command line

option it will build the database automatically, useful for scheduling weekly updates

You can download it from http://support.citrix.com/article/CTX110629

STRESSPRINTERS: STRESSING PRINTER AUTOCREATION

Printer drivers are great source of crash dumps especially in Citrix and Microsoft

terminal services environments Bad printer drivers crash or hang spooler service

(spoolsv.exe) when multiple users connect to a server

Most of bad drivers were designed and implemented for use in a single

user environment without considering multithreading in mind Some bad drivers display

a dialog box every time the printer is created and because this is done on a server side

users cannot dismiss it unless spooler service is configured to interact with the desktop

and an administrator sees the dialog box Some drivers are linked to a debug run-time

library and every exception brings up a dialog effectively hanging the thread and

some-times the whole spooler service if there was heap corruption, for example

Therefore before allowing terminal services users to use certain printers it is

good to simulate multiple users trying to create particular printers to determine bad

drivers and other printer components Originally Citrix had very popular command line

AddPrinter tool for this purpose and it has been replaced by StressPrinters tool where I

designed and implemented GUI to set various options, coordination of multiple

AddPrinter command line tools launched simultaneously with different parameters and

overall log file management We can even export settings to a file and import it on

another server The tool also has 64-bit executables to test printer autocreation on x64

Windows

The tool detects spooler crashes (if spoolsv.exe suddenly disappears from a

process list) so we can check for crash dumps saved if we set up a default postmortem

debugger (Dr Watson or NTSD) If we see the progress bar hanging for a long time then

we can dump the spooler service using Microsoft userdump.exe to check for any stuck

threads and resource contention

You can read documentation and download this tool from Citrix support:

StressPrinters for 32-bit and 64-bit platforms

http://support.citrix.com/article/CTX109374

InstantDump (JIT Process Dumper) 639

INSTANTDUMP (JIT PROCESS DUMPER)

Techniques utilizing user mode process dumpers and debuggers like Microsoft

userdump.exe, NTSD or WinDbg and CDB from Debugging Tools for Windows are too

slow to pick up a process and dump it We need either to attach a debugger

ma-nually, run the command line prompt or switch to Task Manager This deficiency was the

primary motivation for me to use JIT (just-in-time) technology for process dumpers The

new tool, InstantDump, will dump a process instantly and non-invasively in a moment

when we need it How does it work? We point to any window and press hot key

InstantDump could be useful to study hang GUI processes or to get several

dumps of the same process during some period of time (CPU spiking case or memory

leak, for example) or just dump the process for the sake of dumping it (for curiosity)

The tool uses tooltips to dynamically display window information

Here is the short user guide:

1 The program will run only on XP/W2K3/Vista (in fact it will not load on

Windows 2000)

2 Run InstantDump.exe on 32-bit system or InstantDump64.exe on x64

Win-dows If we attempt to run InstantDump.exe on x64 Windows it will show this message

box and quit:

3 InstantDump puts itself into task bar icon notification area:

4 By default, when we move the mouse pointer over windows, the tooltip

follows the cursor describing the process and thread id and process image path (we can

disable tips in Options dialog box):

5 If we hold Ctrl-RightShift-Break for less than a second then the process (which

window is under the cursor) will be dumped according to the settings for external

process dumper in the options dialog (accessible via task bar icon right mouse click):

The saved dump name will be (in our Calculator window case):

calc.exe_9f8(2552)_22-17-56_18-Feb-2007.dmp

There is no NTSD in Vista so we have to use another user mode dumper, for

example, install Microsoft userdump.exe and specify the following command line in

Options dialog:

userdump.exe %d %s

or resort to WinDbg or CDB command line

The tool can be downloaded from here:

http://www.dumpanalysis.org/downloads/InstantDump.zip

TestDefaultDebugger 641

TESTDEFAULTDEBUGGER

It often happens that support engineers advise customers to change their default

postmortem debugger to NTSD But there is no way to test new settings unless an

application crashes again And some customers come back saying that dump files are

not saved despite the new settings and we don’t know whether dump files were not

saved because a crash hadn’t yet happened or default debugger hadn’t been configured

properly or something else had happened

In addition, the arrival of 64-bit Windows brings another problem: there are 2

de-fault postmortem debuggers on 64-bit Windows, one for 32-bit and one for 64-bit

applications respectively (page 633)

The new tool TestDefaultDebugger forces a crash on itself to test the presence

and configuration of a default postmortem debugger (Dr Watson, NTSD or other) Then

if the default postmortem debugger is configured properly OS will launch it to save a

memory dump of TestDefaultDebugger.exe process

If we enabled NTSD as a default postmortem debugger (see

http://support.citrix.com/article/CTX105888) the following console window would

briefly appear:

Postmortem debuggers are explained on page 28

On 64-bit Windows we can run both 32-bit TestDefaultDebugger.exe and 64-bit

TestDefaultDebugger64.exe applications and then open crash dumps to see whether

both postmortem debuggers have been configured properly The tool has also

com-mand line interface so we can use it remotely:

c:\>TestDefaultDebugger.exe now

We can download the tool from Citrix support web site:

TestDefaultDebugger v1.0 for 32-bit and 64-bit platforms

http://support.citrix.com/article/CTX111901

DumpAlerts 643

DUMPALERTS

The tool monitors folders where dumps can be saved including Dr Watson, a

folder specified when NTSD is set as a default debugger and so on It then alerts a user,

an administrator or a software vendor whenever a new dump file is saved:

Icon in System Tray changes its color from green to red

Popup window appears until dismissed

E-mail is sent to a specified address

Sound is played

Custom action is executed, for example, automatically launching WinDbg.exe

with the latest memory dump file or copying it to an ftp server All actions are fully configurable and can be enabled/disabled Here is the screen-

shot of the main window:

DUMPDEPENDS

There are many cases where we need to dump several processes simultaneously

and complete memory dump is not an option DumpDepends tool dumps processes and

optionally package them into a CAB file There are several options:

Dump all processes

Dump important services (Terminal, IMA, CTXXMLSS, Printing, Spooler,

SVCHOST) Dump all processes from the given session ID (additionally including children

and important services if needed) Dump an individual process (optionally including children and important ser-

vices)

The tool will use external process dumpers in noninvasive manner (NTSD by

de-fault or any other specified, like userdump.exe) On x64 it will distinguish between

32-bit and 64-32-bit processes and dump them accordingly Command line option will also be

available

Dump Monitor Suite 645

DUMP MONITOR SUITE

Following the announced Troubleshooting Tool Ideas database Ramzy Mansour

from Citrix Technical Support came up with a brilliant idea about Dump Monitor Suite

and its two useful components for Citrix and Microsoft administrators:

DumpStats:

- Monitors and displays a graphical chart showing which services and processes

crashed or hanged on an individual server, their crash time and date, dump

loca-tion, dump type, crash signature and modules where crashes happened

- Aggregates and displays statistics for the whole server farm

DumpAlerts:

- Sends an e-mail alert and/or an SMS message to a cell phone when any crash or

hang happens

- Configures alerts based on severity and specific processes

Additionally Dump Monitor Suite includes the following components (some of

them already exist):

DumpChecks:

- Enhanced and improved version of Citrix DumpCheck Explorer extension and its

command line version

DumpProperties:

- New Explorer extension (Properties dialog) which shows various data extracted

from a dump, like process name, module list, whether heap checking was enabled

and module name where crash happened

DumpDepends:

- Integrated and enhanced version of SystemDump which allows to

dump dependent processes