I thought that before I talk about how to analyze minidumps I should probably spend some time talking about what they are, how you get them and what can you do with them. I’m going to keep this fairly simple and high level, so those of you who already have a good understanding of crash dumps might want to skim this post. If anyone is interested in more detail they can post a comment and I’ll elaborate where I can.
The easiest way to describe a minidump is that it is a snapshot of the state of an application at a given point in time. They can be capture automatically or manually and depending on the problem you are trying to solve they can be captured at different points in time. If an application is crashing unexpectedly then the minidump will probably (or rather hopefully!) be captured automatically when the crash occurs. Crashing applications can also create their own crash dumps e.g. the Firefox crash reporter. If an application is making excessive use of the cpu then you would normally capture one manually while the excessive usage is occurring. The same strategy would work if you’re trying to solve a problem with excessive memory usage. Solving memory leaks can sometimes be a bit more complicated: you can either wait for the leak to consume enough memory and then capture a minidump, or you’ll have to create a few crash dumps at regular intervals and then look for differences within them.
I found the easiest way to capture minidumps manually was to use WinDbg (part of the Debugging Tools for Windows). The Debugging Tools also include command line applications, but I never looked at them. Once you’ve got the tools installed simply open WinDbg, attach to the offending process (selecting the “Non Invasive” option) and running the dump command e.g. “.dump /map c:\Minidump1.hdmp” (I’ll talk about the “/map” option later). Some things to note here:
- Once you attach WinDbg to your process, the process will be paused. So if you’re doing this in a production environment it’s best to be as quick as possible 😉 Even so, it’s still a pretty quick process (mine used to take around half a minute to create a 600MB crash dump) and is well worth it if it means you can solve a critical problem.
- The tools need to be installed on the machine where the minidump is going to be captured. The installation doesn’t “mess” with your system, but depending on how open minded your sys admin is you might have problems here.
- Using the “/map” option will result in a pretty big file on disk (mine used to be around 600MB). More on this later though
Don’t forget to “Detach” afterwards! That’s “Detach” not “Stop Debugging”!
So WinDbg is a nice and easy way to capture a crash dump manually (and a very powerful tool to analyze it!). But what happens when your process crashes unexpectedly? There are actually a few options here, some dependent on what version of Windows you’re running.
Besides WinDbg (and a few other tools), the Debugging Tools for Windows also includes a script you can use to automatically collect crash dumps when your application crashes. It’s called ADPlus. I’ve never used it, so I can’t say much about it, but I have heard about it quite often. Having read that link I see it’s actually been rewritten recently. I’ll have to find out more about that, could prove interesting.
I relied on Dr Watson (we were using Windows 2003 – old school!) to capture crash dumps when our process crashed. Those crash dumps used to be saved somewhere in the “c:\Windows\PCHEALTH” folder (inside .CAB files if I remember correctly, go have a look at yours, you might be surprised to see what’s there).
In the more recent versions of Windows Dr Watson has been replaced with Windows Error Reporting (WER). I haven’t used this much yet, but it seems like saving crash dumps is actually disabled by default. If you want to change this, then read this article on MSDN.
These days you can even create a crash dump right from the Task Manager: just right click a process and select “Create dump file”. You don’t get to select what information to include in the file though.
Before I talk about what’s inside minidumps here’s a quick exercise for you. Open a command prompt and browse to “%appdata%”, then run “dir *.dmp /s”. I get 14 results, mostly from Firefox crash reports. Those are crash dumps! If you have ones from Firefox do you notice how small they are? I don’t have any that even reach 500KB. Now try running that same “dir *.dmp /s” command in your windows folder. I find a 500MB file called “memory.dmp” in my “c:\Windows” folder. You can try using “dir *.hdmp /s” also, you might get some hits there.
You can see from the samples I found on my local drive that the size of crash dumps can vary dramatically: anything from 500KB to 500MB in my case. This is because minidumps can store various different sets of information. The more information you store the bigger they get. The Firefox ones are quite small, so they probably just store enough information to reconstruct a call stack of running threads. The “memory.dmp” one probably contains the full user space memory of the process, loaded modules, thread information, etc. I’m just guessing here, but in the next few posts I’ll be releasing some code that will allow us to look inside those crash dumps and see what information they contain (as well as some code to capture your own).
When I mention using WinDbg earlier to manually capture a crash dump I said to run the “.dump” command with “/map” as its options, this is what those options mean (from the “Debugging Help” file):
- m – Create a minidump
- a – Creates a minidump with all optional additions. The /ma option is equivalent to /mfFhut — it adds full memory data, handle data, unloaded module information, basic memory information, and thread time information to the minidump. Any failure to read inaccessable memory results in termination of the minidump generation.
- p – Adds process environment block (PEB) and thread environment block (TEB) data to the minidump. This can be useful if you need access to Windows system information regarding the application’s processes and threads.
So I’m basically saying: create a crash dump of everything, or at least very close to everything 🙂
I’m not going to go into the detail of what all of the options are: if you want to know what they are you can open the “Debugging Help” file that gets installed with the Debugging Tools and browse to Windows Debugging\Debugger Reference\Debugger Commands\Meta-Commands in the contents. There is a lot of useful information in that help file.
What I do want to point out though is the description of the “a” option and just how many different pieces of information can be stored in a minidump file, and that’s not even the full list.
In the next post I’ll start talking about how we can use MiniDumpWriteDump to create our own crash dumps with whatever information we want (with some actual code!) and then how we can start reading them (with even more code!)