Tài liệu Memory Dump Analysis Anthology- P20 pdf

Stack Trace Backtrace 573 STACK TRACE BACKTRACE Displaying thread stack trace is the most used action in crash or core dump analy-sis and debugging... C:\MinGW\examples>..\bin\gcc -g -

Disassembler 571 The second GDB command is x/[N]i address where N is the number of instruc-

tions to disassemble:

(gdb) x/i 0x4012f0

0x4012f0 <main>: push ebp

(gdb) x/2i 0x4012f0

0x4012f0 <main>: push ebp

0x4012f1 <main+1>: mov ebp,esp

(gdb) x/3i 0x4012f0

0x4012f0 <main>: push ebp

0x4012f1 <main+1>: mov ebp,esp

0x4012f3 <main+3>: sub esp,0x8

(gdb) x/4i $pc

0x4012f6 <main+6>: and esp,0xfffffff0

0x4012f9 <main+9>: mov eax,0x0

0x4012fe <main+14>: add eax,0xf

0x401301 <main+17>: add eax,0xf

(gdb)

It seems to be no way to disassemble just N instructions in WinDbg However in

WinDbg we can disassemble backwards (ub) This is useful, for example, if we have a

return address and we want to see the CALL instruction:

7d4e9919 lea eax,[ebp+8]

7d4e991c push eax

7d4e991d push 9

7d4e991f push 0FFFFFFFEh

7d4e9921 call dword ptr [kernel32!_imp NtSetInformationThread

(7d4d032c)]

7d4e9927 call dword ptr [ebp+8]

Our next version of the map contains these new commands:

Action | GDB | WinDbg

-

Start the process | run | g

Exit | (q)uit | q

Disassemble (forward) | (disas)semble | uf, u

Disassemble N instructions | x/i | -

Disassemble (backward) | - | ub

Stack Trace (Backtrace) 573

STACK TRACE (BACKTRACE)

Displaying thread stack trace is the most used action in crash or core dump

analy-sis and debugging To show various available GDB commands I created the next version

of the test program with the following source code:

#include <stdio.h>

void func_1(int param_1, char param_2, int *param_3, char *param_4);

void func_2(int param_1, char param_2, int *param_3, char *param_4);

void func_3(int param_1, char param_2, int *param_3, char *param_4);

void func_4();

int val_1;

char val_2;

int *pval_1 = &val_1;

char *pval_2 = &val_2;

void func_4()

{

puts("Hello World!");

}

We need to compile it with -g gcc compiler option to generate symbolic

informa-tion It is needed for GDB to display function arguments and local variables

C:\MinGW\examples> \bin\gcc -g -o test.exe test.c

If you have a crash in func_4 then we can examine stack trace (backtrace) once

we open a core dump Because we don’t have a core dump of our test program we will

simulate the stack trace by putting a breakpoint on func_4 In GDB this can be done by

(103c.17d8): Break instruction exception - code 80000003 (first chance)

eax=7d600000 ebx=7efde000 ecx=00000005 edx=00000020 esi=7d6a01f4

Stack Trace (Backtrace) 575

0:000> g

ModLoad: 71c20000 71c32000 C:\W2K3\SysWOW64\tsappcmp.dll

ModLoad: 77ba0000 77bfa000 C:\W2K3\syswow64\msvcrt.dll

ModLoad: 77f50000 77fec000 C:\W2K3\syswow64\ADVAPI32.dll

ModLoad: 7da20000 7db00000 C:\W2K3\syswow64\RPCRT4.dll

00408be0 55 push ebp

We have to disable optimization in the project properties otherwise Visual C++

compiler optimizes away all function calls and produces the following short code:

0:000> uf main

00401000 push offset test!`string' (004020f4)

00401005 mov dword ptr [test!val_1 (0040337c)],4

0040100f mov byte ptr [test!val_2 (00403378)],64h

00401016 call dword ptr [test!_imp puts (004020a0)]

0040101c add esp,4

0040101f xor eax,eax

00401021 ret

Now we are going to concentrate on commands that examine a call

stack backtrace or bt command shows stack trace backtrace <N> or bt <N> shows only

the innermost N stack frames backtrace -<N> or bt -<N> shows only the outermost N

stack frames backtrace full or bt full additionally shows local variables There are also

variants backtrace full <N> or bt full <N> and backtrace full -<N> or bt full -<N>:

Stack Trace (Backtrace) 577

In WinDbg there is only one k command but it has many parameters, for

Full stack trace without source code lines showing 3 stack arguments for every

stack frame, calling convention and optimization information:

0:000> kvL

ChildEBP RetAddr Args to Child

0012ff20 00408c30 0012ff40 00408c69 00000001 test!func_4 (CONV: cdecl)

0012ff70 00401368 00000001 004230e0 00423120 test!main+0x33 (CONV: cdecl)

0012ffc0 7d4e992a 00000000 00000000 7efde000 test! tmainCRTStartup+0x15f

(FPO: [Non-Fpo]) (CONV: cdecl)

0012fff0 00000000 004013bf 00000000 00000000

kernel32!BaseProcessStart+0x28 (FPO: [Non-Fpo])

Stack trace without source code lines showing all function parameters:

Stack Trace (Backtrace) 579

Stack trace without source code lines showing the innermost 2 frames:

Therefore, the next version of the map contains these new commands:

Action | GDB | WinDbg

-

Start the process | run | g

Exit | (q)uit | q

Disassemble (forward) | (disas)semble | uf, u

Disassemble N instructions | x/<N>i | -

Disassemble (backward) | - | ub

Stack trace | backtrace (bt) | k

Full stack trace | bt full | kv

Partial trace (innermost) | bt <N> | k <N>

Partial trace (outermost) | bt -<N> | -

Stack trace for all threads | thread apply all bt | ~*k

Breakpoint | break | bp

Local Variables 581

LOCAL VARIABLES

Once we get backtrace in GDB or stack trace in WinDbg we are interested in

con-crete stack frames, their arguments and local variables I slightly modified the program

used in the previous part to include some local variables:

#include <stdio.h>

void func_1(int param_1, char param_2, int *param_3, char *param_4);

void func_2(int param_1, char param_2, int *param_3, char *param_4);

void func_3(int param_1, char param_2, int *param_3, char *param_4);

void func_4();

int g_val_1;

char g_val_2;

int *g_pval_1 = &g_val_1;

char *g_pval_2 = &g_val_2;

void func_2(int param_1, char param_2, int *param_3, char *param_4)

In GDB the frame command is used to set the current stack frame Then info args

command can be used to list function arguments and info locals command can be used

to list local variables:

(gdb) break func_4

Breakpoint 1 at 0x401455: file test.c, line 61

(gdb) run

Starting program: C:\MinGW\examples/test.exe

Breakpoint 1, func_4 () at test.c:61

In WinDbg kn command shows stack trace with frame numbers, knL command

additionally omits source code references, frame command switches to particular stack

frame, dv command shows parameters and local variables together, dv /i command

classifies them into categories, parameters or locals, dv /V command shows their

ad-dresses and offsets for the relevant base frame register, usually EBP, dv /t command

shows type information:

Microsoft (R) Windows Debugger Version 6.7.0005.1

Copyright (c) Microsoft Corporation All rights reserved

Local Variables 585

(e38.ac0): Break instruction exception - code 80000003 (first chance)

eax=7d600000 ebx=7efde000 ecx=00000005 edx=00000020 esi=7d6a01f4

ModLoad: 77ba0000 77bfa000 C:\W2K3\syswow64\msvcrt.dll

ModLoad: 00410000 004ab000 C:\W2K3\syswow64\ADVAPI32.dll

ModLoad: 7da20000 7db00000 C:\W2K3\syswow64\RPCRT4.dll

prv param 0012ff20 @ebp+0x08 param_1 = 3

prv param 0012ff24 @ebp+0x0c param_2 = 51 '3'

prv param 0012ff28 @ebp+0x10 param_3 = 0x0012ff30

prv param 0012ff2c @ebp+0x14 param_4 = 0x0040c9d4 "f"

prv local 0012ff14 @ebp-0x04 local_3 = 3

prv local 0012ff68 @ebp-0x08 local_0 = 0

prv local 0012ff6c @ebp-0x04 hello = 0x0040a274 "Hello World!"

0:000> dv /t

int local_0 = 0

char * hello = 0x0040a274 "Hello World!"

Disassemble (forward) | (disas)semble | uf, u

Disassemble N instructions | x/<N>i | -

Disassemble (backward) | - | ub

Stack trace | backtrace (bt) | k

Full stack trace | bt full | kv

Stack trace with parameters | bt full | kP

Partial trace (innermost) | bt <N> | k <N>

Partial trace (outermost) | bt -<N> | -

Stack trace for all threads | thread apply all bt | ~*k

Breakpoint | break | bp

Frame numbers | any bt command | kn

Select frame | frame | frame

Display parameters | info args | dv /t /i /V

Display locals | info locals | dv /t /i /V

Four Pillars 589 PART 8: SOFTWARE TROUBLESHOOTING

FOUR PILLARS

They are (sorted alphabetically):

1 Crash Dump Analysis (also called Memory Dump Analysis or Core Dump

Analysis)

2 Problem Reproduction

3 Trace and Log Analysis

4 Virtual Assistance (also called Remote Assistance)

FIVE GOLDEN RULES

It is difficult to analyze a problem when we have crash dumps and/or

traces from various tracing tools and supporting information we have is incomplete or

missing I came up with this easy to remember 4WS questions to ask when sending or

requesting traces and memory dumps:

What - What had happened or had been observed? Crash or hang, for example?

When - When did the problem happen if traces were recorded for hours?

Where - What server or workstation had been used for tracing or where memory

dumps came from? For example, one trace is from a primary server and two others are

from backup servers or one trace is from a client workstation and the other is from a

server

Why - Why did a customer or a support engineer request a dump file or a trace?

This could shed the light on various assumptions including presuppositions hidden in

problem description

Supporting information - needed to find a needle in a hay: process id, thread id,

etc Also, the answer to the following question is important: how memory dumps and

traces were created?

Every trace or memory dump shall be accompanied by 4WS answers

4WS rule can be applied to any troubleshooting because even the problem

description itself is some kind of a trace

Critical Thinking 591

CRITICAL THINKING

Faulty thinking happens all the time in technical support environments partly due

to hectic and demanding business realities

There is an interesting website that taxonomically organizes fallacies:

http://www.fallacyfiles.org/taxonomy.html

Take, for example, False Cause fallacy Technical examples might include false

causes inferred from trace analysis, customer problem description that includes steps to

reproduce the problem, and so on This also applies to debugging and importance of

critical thinking skills has been emphasized in the following book:

Debugging by Thinking: A Multidisciplinary Approach

Surface-level of basic crash dump analysis is less influenced by false cause

falla-cies because it doesn’t have explicitly recorded sequence of events although some

cau-tion should be exercised during detailed analysis of thread waiting times and other

historical information

TROUBLESHOOTING AS DDEBUGGING

This post is motivated by TRAFFIC steps introduced by Andreas Zeller in his book

“Why Programs Fail?” This book is wonderful and it gives practical debugging skills

co-herent and solid systematical foundation

However these steps are for fixing defects in code, the traditional view of the

software debugging process Based on an analogy with systems theories where we

have different levels of abstraction like psychology, biology, chemistry and physics, I

would say that debugging starts when we have a failure at the system level

If we compare systems to applications, troubleshooting to source code

debug-ging, the question we ask at the higher level is “Who caused the product to fail?” which

also has a business and political flavor Therefore I propose a different acronym:

VERSION If you always try to fix system problems at the code level you will get a huge

“traffic” in all sense but if you troubleshoot them first you get a different system /

subsystem / component version and get your problem solved faster This is why we have

technical support departments in organizations

There are some parallels between TRAFFIC and VERSION steps:

Track View the problem

Reproduce Environment/repro steps

Automate (and simplify) Relevant description

Find origins Subsystem/component

Troubleshooting doesn’t eliminate the need to look at source code In many

cases a support engineer has to be proficient in code reading skill to be able to map

from traces to source code This will help in component identification, especially if the

product has extensive tracing facility

Pooltags 593 PART 9: CITRIX

POOLTAGS

Citrix drivers have their own pooltags Please refer to the following article:

http://support.citrix.com/article/ctx115257

When we see the following or similar output from !poolused WinDbg command

we can update pooltag.txt file located in Debugging Tools for Windows installation

triage folder:

WD UNKNOWN pooltag 'WD ', please update pooltag.txt

Note: ‘Ica’ pooltag doesn’t belong to Citrix drivers although it sounds like “ICA

protocol” It comes from Microsoft termdd.sys driver

THE LIST OF CITRIX SERVICES

In kernel or complete memory dumps coming from Windows servers running

Citrix Presentation Server 4.x we might see the following processes running in session 0,

The List of Citrix Services 595

PROCESS 88fce020 SessionId: 0 Cid: 1204 Peb: 7ffdf000 ParentCid:

01a8

DirBase: 0a43d900 ObjectTable: 88fcfac8 HandleCount: 186

Image: ctxwmisvc.exe

These are Citrix services and the following Citrix article describes them briefly:

Citrix Presentation Server Services Overview

http://support.citrix.com/article/CTX114669

REVERSE ENGINEERING CITRIX THINWIRE

Crash dumps (and live debugging) can be very useful for reverse engineering

component dependencies Let’s look at Microsoft Video Driver Architecture UML

component diagram:

Kernel Mode/

SpaceUser Mode/Space

Coupled with this understanding and armed with Citrix symbol files (which are

freely downloadable from Citrix support) I was able to transform this thread stack below

and other similar stacks into the following UML component diagram (some functions are

shown as module!xxx and offsets are removed for clarity):

Reverse Engineering Citrix ThinWire 597

Then we replace Microsoft components with Citrix ones:

Video Display with vdtw30.dll

Video Miniport with icacdd.sys

Hardware and HAL with Terminal Services stack components (Microsoft

termdd.sys, Citrix wdica.sys, etc)

Kernel Mode/

SpaceUser Mode/Space

Application gdi32.dll user32.dll

Memory Visualization 599 PART 10: SECURITY

MEMORY VISUALIZATION

This security warning is related to sound files and pictures produced by

Dump2Wave (page 521), Dump2Picture (page 532) and WinDbg scripts based on them

These tools do not change computer memory data and it is present in resulting

WAV and BMP files unmodified Do not publish these files on Internet, otherwise

you might expose your private or sensitive data

If you use lossy compression afterwards, like MP3 or JPEG, all original

com-puter memory content is lost and becomes non-recoverable

Therefore, if you create a piece of modern art using Dump2Picture and want to

publish it electronically always transform it into JPEG, for example