Managed Code Exception 331 MANAGED CODE EXCEPTION .NET programs also crash either from defects in .NET runtime (Common Lan- guage Runtime, CLR) or from non-handled runtime exceptions in managed code ex- ecuted by .NET virtual machine. The latter exceptions are re-thrown from .NET runtime to be handled by operating system and intercepted by native debuggers. Therefore our next crash dump analysis pattern is called Managed Code Exception. When we get a crash dump from .NET application it is the dump from a native process. !analyze -v output can usually tell us that exception is actually CLR exception and give us other hints to look at managed code stack (CLR stack): FAULTING_IP: kernel32!RaiseException+53 77e4bee7 5e pop esi EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 77e4bee7 (kernel32!RaiseException+0x00000053) ExceptionCode: e0434f4d (CLR exception) ExceptionFlags: 00000001 NumberParameters: 1 Parameter[0]: 80131604 DEFAULT_BUCKET_ID: CLR_EXCEPTION PROCESS_NAME: mmc.exe ERROR_CODE: (NTSTATUS) 0xe0434f4d - <Unable to get error code text> MANAGED_STACK: !dumpstack -EE No export dumpstack found STACK_TEXT: 05faf3d8 79f97065 e0434f4d 00000001 00000001 kernel32!RaiseException+0x53 WARNING: Stack unwind information not available. Following frames may be wrong. 05faf438 7a0945a4 023f31e0 00000000 00000000 mscorwks!DllCanUnloadNowInternal+0×37a9 05faf4fc 00f2f00a 02066be4 02085ee8 023d0df0 mscorwks!CorLaunchApplication+0×12005 05faf500 02066be4 02085ee8 023d0df0 023d0e2c 0xf2f00a 05faf504 02085ee8 023d0df0 023d0e2c 05e00dfa 0×2066be4 05faf508 023d0df0 023d0e2c 05e00dfa 023d0e10 0×2085ee8 05faf50c 023d0e2c 05e00dfa 023d0e10 05351d30 0×23d0df0 05faf510 05e00dfa 023d0e10 05351d30 023d0e10 0×23d0e2c Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 332 PART 3: Crash Dump Analysis Patterns FOLLOWUP_IP: mscorwks!DllCanUnloadNowInternal+37a9 79f97065 c745fcfeffffff mov dword ptr [ebp-4],0FFFFFFFEh SYMBOL_NAME: mscorwks!DllCanUnloadNowInternal+37a9 MODULE_NAME: mscorwks IMAGE_NAME: mscorwks.dll PRIMARY_PROBLEM_CLASS: CLR_EXCEPTION BUGCHECK_STR: APPLICATION_FAULT_CLR_EXCEPTION Sometimes we can see mscorwks.dll on raw stack or see it loaded and can find it on other thread stacks than the current one. When we get such hints we might want to get managed code stack as well. First we need to load the appropriate WinDbg SOS extension (Son of Strike) corresponding to .NET runtime version. This can be done by the following command: 0:015> .loadby sos mscorwks We can check which SOS extension version was loaded by using .chain command: 0:015> .chain Extension DLL search Path: . . . Extension DLL chain: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.42, API 1.0.0, built Fri Sep 23 08:27:26 2005 [path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll] dbghelp: image 6.6.0007.5, API 6.0.6, built Sat Jul 08 21:11:32 2006 [path: C:\Program Files\Debugging Tools for Windows\dbghelp.dll] ext: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:10:52 2006 [path: C:\Program Files\Debugging Tools for Windows\winext\ext.dll] exts: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:10:48 2006 [path: C:\Program Files\Debugging Tools for Windows\WINXP\exts.dll] uext: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:11:02 2006 [path: C:\Program Files\Debugging Tools for Windows\winext\uext.dll] ntsdexts: image 6.0.5457.0, API 1.0.0, built Sat Jul 08 21:29:38 2006 [path: C:\Program Files\Debugging Tools for Windows\WINXP\ntsdexts.dll] Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Managed Code Exception 333 Then we can use !dumpstack to dump the current stack or !EEStack command to dump all thread stacks. The native stack trace would be mixed with managed stack trace: 0:015> !dumpstack OS Thread Id: 0x16e8 (15) Current frame: kernel32!RaiseException+0x53 ChildEBP RetAddr Caller,Callee 05faf390 77e4bee7 kernel32!RaiseException+0x53, calling ntdll!RtlRaiseException 05faf3a8 79e814da mscorwks!Binder::RawGetClass+0x23, calling mscorwks!Module::LookupTypeDef 05faf3bc 79e87ff4 mscorwks!Binder::IsClass+0x21, calling mscorwks!Binder::RawGetClass 05faf3c8 79f958b8 mscorwks!Binder::IsException+0x13, calling mscorwks!Binder::IsClass 05faf3d8 79f97065 mscorwks!RaiseTheExceptionInternalOnly+0x226, calling kernel32!RaiseException 05faf438 7a0945a4 mscorwks!JIT_Throw+0xd0, calling mscorwks!RaiseTheExceptionInternalOnly 05faf4ac 7a0944ea mscorwks!JIT_Throw+0x1e, calling mscorwks!LazyMachStateCaptureState 05faf4c8 793d424e (MethodDesc 0x7924ad68 +0x2e System.Threading.WaitHandle.WaitOne(Int64, Boolean)), calling mscorwks!WaitHandleNative::CorWaitOneNative 05faf4fc 00f2f00a (MethodDesc 0x4f97500 +0x9a Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[])), calling mscorwks!JIT_Throw 05faf510 05e00dfa (MethodDesc 0×4f98fd8 +0xca MyNamespace.MyClass.MyMethod(Boolean)), calling 05fc7124 05faf55c 00f62fbc (MethodDesc 0×4f95f90 +0×16f4 MyNamespace.MyClass.MyMethod.Initialise(System.Object)) 05faf740 793d912f (MethodDesc 0×7925fc70 +0×2f System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Objec t)) 05faf748 793683dd (MethodDesc 0×7913f3d0 +0×81 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)) 05faf75c 793d9218 (MethodDesc 0×7925fc80 +0×6c System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object )), calling (MethodDesc 0×7913f3d0 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)) 05faf774 79e88f63 mscorwks!CallDescrWorker+0×33 05faf784 79e88ee4 mscorwks!CallDescrWorkerWithHandler+0xa3, calling mscorwks!CallDescrWorker 05faf804 79f20212 mscorwks!DispatchCallBody+0×1e, calling mscorwks!CallDescrWorkerWithHandler 05faf824 79f201bc mscorwks!DispatchCallDebuggerWrapper+0×3d, calling mscorwks!DispatchCallBody 05faf888 79f2024b mscorwks!DispatchCallNoEH+0×51, calling mscorwks!DispatchCallDebuggerWrapper Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 334 PART 3: Crash Dump Analysis Patterns 05faf8bc 7a07bdf0 mscorwks!Holder,2>::~Holder,2>+0xbb, calling mscorwks!DispatchCallNoEH 05faf90c 77e61d1e kernel32!WaitForSingleObjectEx+0xac, calling ntdll!ZwWaitForSingleObject 05faf91c 79ecb4a4 mscorwks!Thread::UserResumeThread+0xfb 05faf92c 79ecb442 mscorwks!Thread::DoADCallBack+0×355, calling mscorwks!Thread::UserResumeThread+0xae 05faf950 79e74afe mscorwks!Thread::EnterRuntimeNoThrow+0×9b, calling mscorwks!_EH_epilog3 05faf988 79e77fe8 mscorwks!PEImage::LoadImage+0×1e1, calling mscorwks!_SEH_epilog4 05faf9c0 79ecb364 mscorwks!Thread::DoADCallBack+0×541, calling mscorwks!Thread::DoADCallBack+0×2a5 05faf9fc 7a0e1b7e mscorwks!Thread::DoADCallBack+0×575, calling mscorwks!Thread::DoADCallBack+0×4d4 05fafa24 7a0e1bab mscorwks!ManagedThreadBase::ThreadPool+0×13, calling mscorwks!Thread::DoADCallBack+0×550 05fafa38 7a07cae8 mscorwks!QueueUserWorkItemCallback+0×9d, calling mscorwks!ManagedThreadBase::ThreadPool 05fafa54 7a07ca48 mscorwks!QueueUserWorkItemCallback, calling mscorwks!UnwindAndContinueRethrowHelperAfterCatch 05fafa90 7a110f08 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0×40 05fafaa8 7a112328 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×1f2, calling mscorwks!ThreadpoolMgr::ExecuteWorkRequest 05fafad0 79e7839d mscorwks!EEHeapFreeInProcessHeap+0×21, calling mscorwks!EEHeapFree 05fafae0 79e782dc mscorwks!operator delete[]+0×30, calling mscorwks!EEHeapFreeInProcessHeap 05fafb14 79ecb00b mscorwks!Thread::intermediateThreadProc+0×49 05fafb48 77e65512 kernel32!FlsSetValue+0xc7, calling kernel32!_SEH_epilog 05fafb6c 75da14d0 sxs!_calloc_crt+0×19, calling sxs!calloc 05fafb80 77e65512 kernel32!FlsSetValue+0xc7, calling kernel32!_SEH_epilog 05fafb88 75da1401 sxs!_CRT_INIT+0×17e, calling sxs!_initptd 05fafb8c 75da1408 sxs!_CRT_INIT+0×185, calling kernel32!GetCurrentThreadId 05fafb9c 30403805 MMCFormsShim!DllMain+0×15, calling MMCFormsShim!PrxDllMain 05fafbb0 30418b69 MMCFormsShim!__DllMainCRTStartup+0×7a, calling MMCFormsShim!DllMain 05fafbdc 75de0e4c sxs!_SxsDllMain+0×87, calling sxs!DllStartup_CrtInit 05fafbf0 30418bf9 MMCFormsShim!__DllMainCRTStartup+0×10a, calling MMCFormsShim!__SEH_epilog4 05fafbf4 30418c22 MMCFormsShim!_DllMainCRTStartup+0×1d, calling MMCFormsShim!__DllMainCRTStartup 05fafbfc 7c81a352 ntdll!LdrpCallInitRoutine+0×14 05fafc24 7c82ee8b ntdll!LdrpInitializeThread+0×1a5, calling ntdll!RtlLeaveCriticalSection 05fafc2c 7c82edec ntdll!LdrpInitializeThread+0×18f, calling ntdll!_SEH_epilog 05fafc7c 7c82ed71 ntdll!LdrpInitializeThread+0xd8, calling ntdll!RtlActivateActivationContextUnsafeFast 05fafc80 7c82ed35 ntdll!LdrpInitializeThread+0×12c, calling ntdll!RtlDeactivateActivationContextUnsafeFast 05fafcb4 7c82edec ntdll!LdrpInitializeThread+0×18f, calling ntdll!_SEH_epilog Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Managed Code Exception 335 05fafcb8 7c827c3b ntdll!NtTestAlert+0xc 05fafcbc 7c82ecb1 ntdll!_LdrpInitialize+0×1de, calling ntdll!_SEH_epilog 05fafd10 7c82ecb1 ntdll!_LdrpInitialize+0×1de, calling ntdll!_SEH_epilog 05fafd14 7c826d9b ntdll!NtContinue+0xc 05fafd18 7c8284da ntdll!KiUserApcDispatcher+0×3a, calling ntdll!NtContinue 05faffa4 79ecaff9 mscorwks!Thread::intermediateThreadProc+0×37, calling mscorwks!_alloca_probe_16 05faffb8 77e64829 kernel32!BaseThreadStart+0×34 .NET language symbolic names are usually reconstructed from .NET assembly metadata. We can examine a CLR exception and get managed stack trace by using !PrintException and !CLRStack commands, for example: 0:014> !PrintException Exception object: 02320314 Exception type: System.Reflection.TargetInvocationException Message: Exception has been thrown by the target of an invocation. InnerException: System.Runtime.InteropServices.COMException, use !PrintException 023201a8 to see more StackTrace (generated): SP IP Function 075AF4FC 016BFD9A Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[]) . . . 075AF740 793D87AF System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Objec t) 075AF748 793608FD System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 075AF760 793D8898 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object ) StackTraceString: <none> HResult: 80131604 0:014> !PrintException 023201a8 Exception object: 023201a8 Exception type: System.Runtime.InteropServices.COMException Message: Error HRESULT E_FAIL has been returned from a call to a COM component. InnerException: <none> StackTrace (generated): SP IP Function 00000000 00000001 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 336 PART 3: Crash Dump Analysis Patterns Ironring.Management.MMC.IMMCFormsShim.HostUserControl3(System.Object, System.Object, System.String, System.String, Int32, Int32) 0007F724 073875B9 Ironring.Management.MMC.FormNode.SetShimControl(System.Object) 0007F738 053D9DDE Ironring.Management.MMC.FormNode.set_ControlType(System.Type) . . . StackTraceString: <none> HResult: 80004005 0:014> !CLRStack OS Thread Id: 0x11ec (14) ESP EIP 075af4fc 016bfd9a Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[]) . . . 075af740 793d87af System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Objec t) 075af748 793608fd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 075af760 793d8898 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object ) 075af8f0 79e7be1b [GCFrame: 075af8f0] !help command gives the list of other available SOS extension commands: 0:014> !help Object Inspection DumpObj (do) DumpArray (da) DumpStackObjects (dso) DumpHeap DumpVC GCRoot ObjSize FinalizeQueue PrintException (pe) TraverseHeap Examining code and stacks Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Managed Code Exception 337 Threads CLRStack IP2MD U DumpStack EEStack GCInfo EHInfo COMState BPMD Examining CLR data structures DumpDomain EEHeap Name2EE SyncBlk DumpMT DumpClass DumpMD Token2EE EEVersion DumpModule ThreadPool DumpAssembly DumpMethodSig DumpRuntimeTypes DumpSig RCWCleanupList DumpIL Diagnostic Utilities VerifyHeap DumpLog FindAppDomain SaveModule GCHandles GCHandleLeaks VMMap VMStat ProcInfo StopOnException (soe) MinidumpMode Other FAQ Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 338 PART 3: Crash Dump Analysis Patterns In the case where .NET CLR runtime is version 1.x we might get messages point- ing to some .NET DLL and this could be the indication that some threads have managed code: *** WARNING: Unable to verify checksum for mscorlib.dll *** ERROR: Module load completed but symbols could not be loaded for mscorlib.dll In some cases we cannot load the appropriate SOS extension automatically: 0:000> .loadby sos mscorwks Unable to find module ―mscorwks‖ Then we can try SOS version 1.0 0:000> !clr10\sos.EEStack Loaded Son of Strike data table version 5 from ―C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll‖ The following message means that the server version of CLR is used: 0:000> .loadby sos mscorwks Unable to find module ―mscorwks‖ 0:000> .loadby sos mscorsvr 0:000> !help SOS : Help For some crash dumps we get the following message saying that sos.dll cannot be found: 0:000> .loadby sos mscorwks The call to LoadLibrary(C:\WIN_NO_SP\Microsoft.NET \Framework\v2.0.50727\sos) failed, Win32 error 0n126 ―The specified module could not be found.‖ Please check your debugger configuration and/or network access Here we need to check where Microsoft.NET\Framework\v2.0.50727\sos.dll is installed on our crash dump analysis host and use .load command: 0:000> .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll The version of WinDbg since 6.8.4.0 and !analyze -v command show both native and managed stack traces from .NET 64-bit application memory dump so there is no need to load SOS manually there. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Managed Code Exception 339 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 340 PART 3: Crash Dump Analysis Patterns TRUNCATED DUMP Sometimes the page file size is less than the amount of physical memory. If this is the case and we have configured “Complete memory dump” in Startup and Recovery settings in Control Panel we get truncated memory dumps. Therefore we can call our next pattern Truncated Dump. WinDbg prints a warning when we open such a dump file: ************************************************************ WARNING: Dump file has been truncated. Data may be missing. ************************************************************ We can double check this with !vm command: kd> !vm *** Virtual Memory Usage *** Physical Memory: 511859 ( 2047436 Kb) Paging File Name paged out Current: 1536000 Kb Free Space: 1522732 Kb Minimum: 1536000 Kb Maximum: 1536000 Kb We see that the page file size is 1.5Gb but the amount of physical memory is 2Gb. When BSOD happens the physical memory contents will be saved to the page file and the dump file size will be no more than 1.5Gb effectively truncating the data needed for crash dump analysis. Sometimes we can still access some data in truncated dumps but we need to pay attention to what WinDbg says. For example, in the truncated dump shown above the stack and driver code are not available: kd> kv ChildEBP RetAddr Args to Child WARNING: Stack unwind information not available. Following frames may be wrong. f408b004 00000000 00000000 00000000 00000000 driver+0x19237 kd> r Last set context: eax=89d55230 ebx=89d21130 ecx=89d21130 edx=89c8cc20 esi=89e24ac0 edi=89c8cc20 eip=f7242237 esp=f408afec ebp=f408b004 iopl=0 nv up ei ng nz ac po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010292 driver+0x19237: f7242237 ?? ??? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 342 PART 3: Crash Dump Analysis Patterns If due to some reasons we cannot increase the size of our page file we just should configure “Kernel memory dump in Startup and Recovery For most all bugchecks kernel memory dump is sufficient except manual crash dumps when we need to inspect user process space Please purchase PDF Split-Merge... to be signaled Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 356 PART 3: Crash Dump Analysis Patterns MEMORY LEAK (PROCESS HEAP) Memory Leak is another pattern that may be finally manifested as Insufficient Memory pattern (page 302) in a crash dump Here I’ll cover process heap memory leaks They are usually identified when the process virtual memory size grows over time It... list of heap segments individual heap allocation entries will be dumped as well This could be very big output and we should open the log file in advance by using logopen command The output can be like this (taken from another dump) : Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 360 PART 3: Crash Dump Analysis Patterns 0:000> !heap -a 000a0000 Segment00 at 000a0000:... we can use !whattime command: 3: kd> !whattime 0n7256 7256 Ticks in Standard Time: 01:53.375s Also !stacks command shows Ticks data for threads Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 348 PART 3: Crash Dump Analysis Patterns DEADLOCK (MIXED OBJECTS) This is another variant of Deadlock pattern (page 276) when we have mixed synchronization objects, for example, events... 01014220 00000000 01014234 00000000 0101e614 00000000 0101e644 00fa840f 00000000 01063afc 00000000 01063b5c 00000000 01065b28 00000000 01065b7c Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 350 PART 3: Crash Dump Analysis Patterns 043412a0 043412a4 0434ab68 0434ab6c 0483fd68 0483fdd8 0483fdec 0483fe08 0483fe20 05296f58 05297eb0 0557cf9c 05580adc 0558715c 0558d3cc 0559363c... 00000000 00000000 00000000 00000000 00800000 00008000 00808000 00000080 00800080 00008080 00808080 00c0c0c0 00ff0000 0000ff00 00ffff00 000000ff Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 352 PART 3: Crash Dump Analysis Patterns 0568ef5c 0568ef60 00ff00ff 0000ffff 0581ff58 0581ff5c 0581ff60 0581ff64 0581ff68 0581ff6c 0581ff70 0581ff74 0581ff78 0581ff7c 0581ff80 0581ff84... dcbabbbb ffffffff c0c00ac1 00000000 c0c0c0c0 c0c0c0c0 c0c0c0c0 c0c0c0c0 c0c0c0c0 c0c0c0c0 ffffffff c0c00ac1 00000000 c0c0c0c0 c0c0c0c0 c0c0c0c0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 354 PART 3: Crash Dump Analysis Patterns We see that the address 0581ff88 is the most meaningful and it also has WaitForSingleObjectEx nearby This address belongs to the raw stack of... WaitListEntry : _LIST_ENTRY [ 0x88091e10 - 0x88029ce0 ] SwapListEntry : _SINGLE_LIST_ENTRY Queue : (null) WaitTime : 0×82de9b KernelApcDisable : 0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 344 PART 3: Crash Dump Analysis Patterns This value is also displayed in the decimal format as Wait Start TickCount when we list threads or use !thread command: 0: kd> ? 0x82de9b Evaluate... process heap is under suspicion here To confirm this we need to sample 2-3 consecutive user memory dumps at process sizes 100Mb, 200Mb and 300Mb, for example This can be done by using Microsoft userdump.exe command line tool Then we can see whether there is any heap growth by using !heap -s WinDbg command: 1st dump 0:000> !heap -s Heap Flags Reserv Commit Virt (k) (k) (k) 00140000... 07f30000 00001002 64 16 16 09050000 00001002 256 12 12 09c80000 00001002 261376 221152 221928 007d0000 00001003 256 192 192 00810000 00001003 256 4 4 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 358 PART 3: Crash Dump Analysis Patterns 0bdd0000 0be10000 0be50000 0be90000 0bed0000 0bf10000 0bf50000 0bf90000 00860000 00870000 0d760000 0dc60000 0c3a0000 0c3d0000 08450000 00001003 . structures DumpDomain EEHeap Name2EE SyncBlk DumpMT DumpClass DumpMD Token2EE EEVersion DumpModule ThreadPool DumpAssembly DumpMethodSig DumpRuntimeTypes DumpSig. extension commands: 0:014> !help Object Inspection DumpObj (do) DumpArray (da) DumpStackObjects (dso) DumpHeap DumpVC GCRoot ObjSize FinalizeQueue PrintException